{ "text": "hey everyone welcome to nning IO I'm JDand today we're going to be looking atreal time in browser speech recognition\nusing web GPU and Transformer JS toactually use speech in the browser sowith that let's go ahead and getstarted all right the first thing that\nwe're going to do everyone is we'regoing to go to this hugging face spaceand basically what we're going to do is\nlook at this demo for the whisper webGPO uh it's real time in browserecognition it's using the whisper basewhich is a 73 million parameter speech\nrecognition mod model and using onx andTransformer JS so you can see that theaudio audio visualizer is already\npicking up my voice from the microphoneand uh able to do some kind of uh visualwhat we're going to do is actually load\nthis model and then we should be able tosee real time uh transcription but whatwe want to showis also that if you go and look at your\nnetwork uh right now I have I'm on thiswasm tab and we want to pay attention toboth Fetch and wasm so we're going to go\nahead and look at the model and you cansee it pulled it in really quickly uhcould be uh because it's cached but uh\nthis was incredibly fast and it'salready doing the the transcription inreal time you can see that it's trying\naddpunctuation and uh the token count downhere if we go over to the wasm tab thisall that was loaded it's going\nthrough a worker to establish this andthe size was 4 megabytes so again thiscrazy fast for doing real time uh\ntranscription in the browser superhelpful if you were trying to do uh likechat based systems where you actually\nwanted to have voice in with your chatas without having to go out to open AIor some otherum closed Source or even open source\nmodel that uh requires a backend callright again we didn't fetch anythingfrom our back end this was just loaded\nthrough the browser so now what we'regoing to do is we're going to take thisand we're going to actually look at the\nrepo and and dig into the code so if youcome over here to Transformers Jsthe thing you want to note is you wanto be on uh the\nV3 Branch this is the the new Branchit's still experimental but it's goingto be the next version of Transformer JS\na lot of great examples we've beentrying to go through some of them itfeels like every day they're coming out\nwith a new demo a new model that they'reable to implement um definitely reallycool stuff so if we look at this example\nso we go into Transformer JS examplesand Whisper then we're able to actuallyD dive into this code so what we're\ngoing to do is we're just going to pullthis down get this running on our localmachine and then we'll start digging\ninto the code so I'm going to use cursorI'm already I already have it pulledown but you can do uh get clone andthen come over\nhere and just dothis uh we are so I'm in the root sowhat we need to do is go toexamples and then we're going to do web\nGPU and go towhisper and we'll do an npminstall and then we'll go ahead and doan npm runDev and what we're going to do is we\nhave our Local Host here so we can gethis up and running we'll watch reallyquickly if we can gethis uh loaded again super quick you saw\nhow quickly the the load time was forthat again some of it sometimes it'scashed and the first one's going to be\nslower but still reallyimpressive so if we look at some of thecore files in here we have our app whichis the main main file and what we're\ngoing to be looking at is how we'reactually pulling in this worker so thisworker. JS is where we're actually going\nto do our Logic for Transformer Js we'repulling in the model we're actuallysending messages back and forth where\nyou can see right here that as messagesare coming being received we're actuallygoing to be looking for the type of\nevent that is being pass and then sendthat uh data as wellso you can see most of this is for likeloading there's uh some update which is\nwhere you're updating the textwe're actually going to look and seewhere this is uh is working so if wecontinue down and we are looking at the\nrecorder this is all the informationabout starting and stopping therecording and then what's Happening Here\nis we are actually looking at how thisinformation is being sent with this codeso righthere it's saying as the recording is\ncoming in we're going to take a blob andchunk it and so then we're going to makea file reader and read those and then\nsend that on when the file is beingread get that bit information that'sdecoded slice it and then send thoseslices over to our worker so we're\nposting the message to generate ourtranscription on the Fly then this isjust our uh our HTML that we're going to\nbe pullingin so what we'll do is we'll jump overto the worker and then we'll take a lookat how this is actually functioning real\nquick everyone if you haven't alreadyplease remember to like And subscribe ithelps more than you know if you have any\nquestions please leave them in thecomments and with that let's get back toit all right so now that we're in our\nworker file we're going to startlooking at how this actually functionsfirst thing is we're loading inTransformers we're going to pull in our\nAutokenizer our Auto processor and then uhwhisper and text stream as well as fullso if we look at what we're first doing\nwe have this class which is our Pipelineand then we're actually using ourinstance where we're getting our model\nso we're saying this is our model ID wehave our tokenizer our processor we'reactually getting the model based on our\nID at this point uh we're telling itwhat our dtype is and then specificallywe're using device webGPU and once all of these have processed\nuh then our git instance has beenestablished if we scroll down to thebottom we can kind of look at like someof the loader functions so you can see\nhere there's the load this happens likewhere we're actually posting backmessages to actually see see theprogress bar things like that we're\nstablishing some events based on ourmessage and then once the the browser uhor once the model is actually loaded\nwe're going to be looking for thisgenerate function this function is whatreally is going to be taking the audio\nand taking the language and thenactually processing it so the firsthing we'll be doing is saying is if ifprocessing return\nif it is not processing then go aheadand set that processing to Truemaintaining State we're actually going\nto send a message back to say that we'retrans that the thread is starting sowe're saying start as our status and\nthen we're actually pulling in ourinstance so again where we set ourtokenizer and our processor and modelwe're actually pulling that from our um\ninstance and we're starting to actuallyprocess so we have our call backfunction which is going to give us the\noutput of our post message of update andinformation about our TPS our tokens andour actual output and then we we have\nour streamer and we this is where we'reactuallyprocessing our information so what we'resaying is our input we're taking our AO\naudio and we're uh putting it throughour processor we then this is ourtranscription our transcription BAS\nbased on our model and our generatewe're saying we're passing our sameinputs our language and we're actually\nstreaming the events or streaming theoutput so now we have our tokenizer thisgives us the information that's\ndisplayed in the bottom right of howmany tokens we're actually using andthen we're posting back that chunk of\ntext as it is complete and so rememberif we are looking at what these postmessages are we're coming back here and\nthat is the information that is being onmessage received so if we look downtoupdate we're seeing that we're setting\nour TPS here as well as uh our messageso our on message right here this iswhere we're being sending theinformation back and then that's how\ne're actually passing information downhere so this is where ourTPS is and then uhthe this the rest of the progress bar in\nthe audio I'm looking fortheactual uh information around the text ohit's right here so on ready we'repassing the information of the text\nhat's beingupdated and that's how this works is webasically are taking this informationfrom after after it passes passes\nthrough the model and then right here sobeing generated by the model and thenposting that information back\nso it's processing the auto chunk it isaudio chunk and it's uh then convertingthat and doing a transcription on this\ngeneration then it's streaming it'sdoing the tokenizer and posting all thatinformation back all right everyone\nthat's it for us today what we coveredwas using real time speech recognitionin the browser with web GPU andTransformers JS we're able to dig\nthrough some of the repo and actuallyget their demo up and running and sowith that happy nerding", "language": "en", "duration": 663.0, "segments": [ { "id": 0, "start": 0.48, "end": 9.28, "text": "hey everyone welcome to nning IO I'm JDand today we're going to be looking atreal time in browser speech recognition", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 1, "start": 9.28, "end": 19.72, "text": "using web GPU and Transformer JS toactually use speech in the browser sowith that let's go ahead and getstarted all right the first thing that", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 2, "start": 19.72, "end": 26.16, "text": "we're going to do everyone is we'regoing to go to this hugging face spaceand basically what we're going to do is", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 3, "start": 26.16, "end": 36.52, "text": "look at this demo for the whisper webGPO uh it's real time in browserecognition it's using the whisper basewhich is a 73 million parameter speech", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 4, "start": 36.52, "end": 45.0, "text": "recognition mod model and using onx andTransformer JS so you can see that theaudio audio visualizer is already", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 5, "start": 45.0, "end": 53.96, "text": "picking up my voice from the microphoneand uh able to do some kind of uh visualwhat we're going to do is actually load", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 6, "start": 53.96, "end": 64.159, "text": "this model and then we should be able tosee real time uh transcription but whatwe want to showis also that if you go and look at your", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 7, "start": 64.159, "end": 71.92, "text": "network uh right now I have I'm on thiswasm tab and we want to pay attention toboth Fetch and wasm so we're going to go", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 8, "start": 71.92, "end": 82.68, "text": "ahead and look at the model and you cansee it pulled it in really quickly uhcould be uh because it's cached but uh", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 9, "start": 82.68, "end": 90.079, "text": "this was incredibly fast and it'salready doing the the transcription inreal time you can see that it's trying", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 10, "start": 90.079, "end": 98.56, "text": "addpunctuation and uh the token count downhere if we go over to the wasm tab thisall that was loaded it's going", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 11, "start": 98.56, "end": 108.04, "text": "through a worker to establish this andthe size was 4 megabytes so again thiscrazy fast for doing real time uh", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 12, "start": 108.04, "end": 116.2, "text": "transcription in the browser superhelpful if you were trying to do uh likechat based systems where you actually", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 13, "start": 116.2, "end": 127.439, "text": "wanted to have voice in with your chatas without having to go out to open AIor some otherum closed Source or even open source", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 14, "start": 127.439, "end": 134.56, "text": "model that uh requires a backend callright again we didn't fetch anythingfrom our back end this was just loaded", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 15, "start": 134.56, "end": 142.68, "text": "through the browser so now what we'regoing to do is we're going to take thisand we're going to actually look at the", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 16, "start": 142.68, "end": 154.4, "text": "repo and and dig into the code so if youcome over here to Transformers Jsthe thing you want to note is you wanto be on uh the", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 17, "start": 154.4, "end": 162.68, "text": "V3 Branch this is the the new Branchit's still experimental but it's goingto be the next version of Transformer JS", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 18, "start": 162.68, "end": 166.92, "text": "a lot of great examples we've beentrying to go through some of them itfeels like every day they're coming out", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 19, "start": 166.92, "end": 176.959, "text": "with a new demo a new model that they'reable to implement um definitely reallycool stuff so if we look at this example", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 20, "start": 176.959, "end": 184.239, "text": "so we go into Transformer JS examplesand Whisper then we're able to actuallyD dive into this code so what we're", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 21, "start": 184.239, "end": 188.959, "text": "going to do is we're just going to pullthis down get this running on our localmachine and then we'll start digging", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 22, "start": 188.959, "end": 200.519, "text": "into the code so I'm going to use cursorI'm already I already have it pulledown but you can do uh get clone andthen come over", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 23, "start": 200.519, "end": 214.64, "text": "here and just dothis uh we are so I'm in the root sowhat we need to do is go toexamples and then we're going to do web", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 24, "start": 214.64, "end": 229.68, "text": "GPU and go towhisper and we'll do an npminstall and then we'll go ahead and doan npm runDev and what we're going to do is we", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 25, "start": 229.68, "end": 240.079, "text": "have our Local Host here so we can gethis up and running we'll watch reallyquickly if we can gethis uh loaded again super quick you saw", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 26, "start": 240.079, "end": 246.4, "text": "how quickly the the load time was forthat again some of it sometimes it'scashed and the first one's going to be", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 27, "start": 246.4, "end": 260.28, "text": "slower but still reallyimpressive so if we look at some of thecore files in here we have our app whichis the main main file and what we're", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 28, "start": 260.28, "end": 267.68, "text": "going to be looking at is how we'reactually pulling in this worker so thisworker. JS is where we're actually going", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 29, "start": 267.68, "end": 274.16, "text": "to do our Logic for Transformer Js we'repulling in the model we're actuallysending messages back and forth where", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 30, "start": 274.16, "end": 284.199, "text": "you can see right here that as messagesare coming being received we're actuallygoing to be looking for the type of", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 31, "start": 284.199, "end": 298.479, "text": "event that is being pass and then sendthat uh data as wellso you can see most of this is for likeloading there's uh some update which is", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 32, "start": 298.479, "end": 312.52, "text": "where you're updating the textwe're actually going to look and seewhere this is uh is working so if wecontinue down and we are looking at the", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 33, "start": 312.52, "end": 319.639, "text": "recorder this is all the informationabout starting and stopping therecording and then what's Happening Here", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 34, "start": 319.639, "end": 331.479, "text": "is we are actually looking at how thisinformation is being sent with this codeso righthere it's saying as the recording is", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 35, "start": 331.479, "end": 341.96, "text": "coming in we're going to take a blob andchunk it and so then we're going to makea file reader and read those and then", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 36, "start": 341.96, "end": 357.4, "text": "send that on when the file is beingread get that bit information that'sdecoded slice it and then send thoseslices over to our worker so we're", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 37, "start": 357.4, "end": 366.039, "text": "posting the message to generate ourtranscription on the Fly then this isjust our uh our HTML that we're going to", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 38, "start": 366.039, "end": 375.599, "text": "be pullingin so what we'll do is we'll jump overto the worker and then we'll take a lookat how this is actually functioning real", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 39, "start": 375.599, "end": 381.319, "text": "quick everyone if you haven't alreadyplease remember to like And subscribe ithelps more than you know if you have any", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 40, "start": 381.319, "end": 388.12, "text": "questions please leave them in thecomments and with that let's get back toit all right so now that we're in our", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 41, "start": 388.12, "end": 396.96, "text": "worker file we're going to startlooking at how this actually functionsfirst thing is we're loading inTransformers we're going to pull in our", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 42, "start": 396.96, "end": 409.0, "text": "Autokenizer our Auto processor and then uhwhisper and text stream as well as fullso if we look at what we're first doing", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 43, "start": 409.0, "end": 415.28, "text": "we have this class which is our Pipelineand then we're actually using ourinstance where we're getting our model", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 44, "start": 415.28, "end": 423.24, "text": "so we're saying this is our model ID wehave our tokenizer our processor we'reactually getting the model based on our", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 45, "start": 423.24, "end": 434.28, "text": "ID at this point uh we're telling itwhat our dtype is and then specificallywe're using device webGPU and once all of these have processed", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 46, "start": 434.28, "end": 444.16, "text": "uh then our git instance has beenestablished if we scroll down to thebottom we can kind of look at like someof the loader functions so you can see", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 47, "start": 444.16, "end": 452.84, "text": "here there's the load this happens likewhere we're actually posting backmessages to actually see see theprogress bar things like that we're", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 48, "start": 452.84, "end": 462.639, "text": "stablishing some events based on ourmessage and then once the the browser uhor once the model is actually loaded", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 49, "start": 462.639, "end": 471.159, "text": "we're going to be looking for thisgenerate function this function is whatreally is going to be taking the audio", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 50, "start": 471.159, "end": 480.12, "text": "and taking the language and thenactually processing it so the firsthing we'll be doing is saying is if ifprocessing return", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 51, "start": 480.12, "end": 486.8, "text": "if it is not processing then go aheadand set that processing to Truemaintaining State we're actually going", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 52, "start": 486.8, "end": 493.0, "text": "to send a message back to say that we'retrans that the thread is starting sowe're saying start as our status and", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 53, "start": 493.0, "end": 501.84, "text": "then we're actually pulling in ourinstance so again where we set ourtokenizer and our processor and modelwe're actually pulling that from our um", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 54, "start": 501.84, "end": 509.199, "text": "instance and we're starting to actuallyprocess so we have our call backfunction which is going to give us the", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 55, "start": 509.199, "end": 520.32, "text": "output of our post message of update andinformation about our TPS our tokens andour actual output and then we we have", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 56, "start": 520.32, "end": 531.48, "text": "our streamer and we this is where we'reactuallyprocessing our information so what we'resaying is our input we're taking our AO", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 57, "start": 531.48, "end": 539.44, "text": "audio and we're uh putting it throughour processor we then this is ourtranscription our transcription BAS", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 58, "start": 539.44, "end": 545.56, "text": "based on our model and our generatewe're saying we're passing our sameinputs our language and we're actually", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 59, "start": 545.56, "end": 552.0, "text": "streaming the events or streaming theoutput so now we have our tokenizer thisgives us the information that's", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 60, "start": 552.0, "end": 559.519, "text": "displayed in the bottom right of howmany tokens we're actually using andthen we're posting back that chunk of", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 61, "start": 559.519, "end": 567.519, "text": "text as it is complete and so rememberif we are looking at what these postmessages are we're coming back here and", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 62, "start": 567.519, "end": 579.24, "text": "that is the information that is being onmessage received so if we look downtoupdate we're seeing that we're setting", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 63, "start": 579.24, "end": 590.04, "text": "our TPS here as well as uh our messageso our on message right here this iswhere we're being sending theinformation back and then that's how", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 64, "start": 590.04, "end": 602.8, "text": "e're actually passing information downhere so this is where ourTPS is and then uhthe this the rest of the progress bar in", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 65, "start": 602.8, "end": 614.399, "text": "the audio I'm looking fortheactual uh information around the text ohit's right here so on ready we'repassing the information of the text", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 66, "start": 614.399, "end": 622.24, "text": "hat's beingupdated and that's how this works is webasically are taking this informationfrom after after it passes passes", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 67, "start": 622.24, "end": 629.92, "text": "through the model and then right here sobeing generated by the model and thenposting that information back", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 68, "start": 629.92, "end": 639.639, "text": "so it's processing the auto chunk it isaudio chunk and it's uh then convertingthat and doing a transcription on this", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 69, "start": 639.639, "end": 646.16, "text": "generation then it's streaming it'sdoing the tokenizer and posting all thatinformation back all right everyone", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 70, "start": 646.16, "end": 657.04, "text": "that's it for us today what we coveredwas using real time speech recognitionin the browser with web GPU andTransformers JS we're able to dig", "source": "video_source_subtitle", "authority": "video_source_subtitle" }, { "id": 71, "start": 657.04, "end": 663.0, "text": "through some of the repo and actuallyget their demo up and running and sowith that happy nerding", "source": "video_source_subtitle", "authority": "video_source_subtitle" } ], "transcript_selection": { "schema": "videonote.transcript_selection.v1", "status": "SELECTED", "requested_language": "en", "duration": 663.0, "reference_path": "/Users/sagawa/AI_WORK/video_notes/url/20260902_112101__How_to_use_Real_Time_In-Browser_Transcription_with_WebGPU_Whisper_and_Transforme/reference_subtitle.vtt", "reference_kind": "automatic_original", "reference_language": "en-orig", "selected_source": "video_source_subtitle", "source_caption_kind": "automatic_original", "source_caption_processing": "rolling_dedup_and_sentence_merge", "youtube_overlap_clamps": [], "general_high_confidence_repairs": [], "decision_reason": "complete_source_subtitle_fast_path_no_whisper", "whisper_rejected": false, "source_profile": { "segment_count": 72, "duration_coverage": 0.999246 } } }