Speech To Text On Client Side Using HTML5 and Streamlit Bokeh Events

[ash2shukla]

Hi Guys,

I just came across this question Audio display and thought to myself what if we want to do the opposite , Its pretty straight forward if you are listening through mic where streamlit server is hosted but it gets a little tricky if you want to do it on client side.
Worry not. Javascript to the rescue,

Checkout this ( buggy ) snippet, that does just this !

import streamlit as st
from bokeh.models.widgets import Button
from bokeh.models import CustomJS
from streamlit_bokeh_events import streamlit_bokeh_events

stt_button = Button(label="Speak", width=100)

stt_button.js_on_event("button_click", CustomJS(code="""
    var recognition = new webkitSpeechRecognition();
    recognition.continuous = true;
    recognition.interimResults = true;
 
    recognition.onresult = function (e) {
        var value = "";
        for (var i = e.resultIndex; i < e.results.length; ++i) {
            if (e.results[i].isFinal) {
                value += e.results[i][0].transcript;
            }
        }
        if ( value != "") {
            document.dispatchEvent(new CustomEvent("GET_TEXT", {detail: value}));
        }
    }
    recognition.start();
    """))

result = streamlit_bokeh_events(
    stt_button,
    events="GET_TEXT",
    key="listen",
    refresh_on_update=False,
    override_height=75,
    debounce_time=0)

if result:
    if "GET_TEXT" in result:
        st.write(result.get("GET_TEXT"))

Hope you guys will find it interesting !

PS. In GIF thats STT of me speaking

[Marisa_Smith]

Hey @ash2shukla!

Thats super cool, so you press this button and then speak into your mic? Do you need an external mic (like on a pair of headphones) or would it work with your computers builtin mic?

(side note: I love how you almost always put gifs in your posts, it makes them so easy to read and understand!)

[ash2shukla]

Hey @Marisa_Smith!

It works with any mic input

( and yay! thanks ! haha )

[lazeni]

Thank you so much that is very cool,
in the same way, could you just convert text to voice without cliking in button ?

[ash2shukla]

Hey @lazeni,

You could do this to do TTS instead of STT,

import streamlit as st
from bokeh.models.widgets import Button
from bokeh.models import CustomJS

text = st.text_input("Say what ?")

tts_button = Button(label="Speak", width=100)

tts_button.js_on_event("button_click", CustomJS(code=f"""
    var u = new SpeechSynthesisUtterance();
    u.text = "{text}";
    u.lang = 'en-US';

    speechSynthesis.speak(u);
    """))

st.bokeh_chart(tts_button)

But the problem lies here as well, you need to click the speak button.

IMHO, it looks a little more neat and it doesn’t have addition gTTS dependencies…

[napoles3d]

Amazing @ash2shukla here is the output on my side

[ash2shukla]

awesome !

[napoles3d]

Hey @ash2shukla, I was wondering if something similar could work for using the webcam. What do you think?

[ash2shukla]

Hey @napoles3d, I haven’t tried it recently but I think webcam and mic won’t work due to sandboxing of components.
The two things I’ve been able to make work were, speech and geolocation APIs…
Let me know if webcam works !

[napoles3d]

mhhh, I see… someone else tried but didn’t work (Big correction, It actually works!! sorry @Luke ) :

Webcam HTML component

Hey everyone, I’m attempting to use a custom html component to access the webcam in a streamlit app. I have the webcam html/js from this codepen, which I’ve refactored into a single html file webcam2.html, and this works fine in the browser (once you have accepted the permissions). [image] Then using streamlit.components.v1.html I am displaying the same html/js inside a streamlit app called webcam.py. [image] And the app running, [image] Streamlit gets pretty close to rendering the who…

[oscar13ud]

Hi, nice work, how can i save the audio file?, i will aprecciate your advice, thanks so much!

[asohn123]

@ash2shukla How do I go about accessing all of the text that was uttered to process it (not just the intermediate outputs)? Thank you!

[Jishnu_Nair]

@ash2shukla I want to implement this app using streamlit in google colab. So please tell me how can I implement this

[ChainYo]

it seems that you can change these two parameters:

recognition.continuous = true;
recognition.interimResults = true;

By changing there state you will be able to avoid intermediate results and continuous recording.

[iliboy]

@ash2shukla

import streamlit as st
from bokeh.models.widgets import Button
from bokeh.models import CustomJS
from streamlit_bokeh_events import streamlit_bokeh_events

stt_button = Button(label="Speak", width=100)

stt_button.js_on_event("button_click", CustomJS(code="""
    var recognition = new webkitSpeechRecognition();
    recognition.continuous = true;
    recognition.interimResults = true;
 
    recognition.onresult = function (e) {
        var value = "";
        for (var i = e.resultIndex; i < e.results.length; ++i) {
            if (e.results[i].isFinal) {
                value += e.results[i][0].transcript;
            }
        }
        if ( value != "") {
            document.dispatchEvent(new CustomEvent("GET_TEXT", {detail: value}));
        }
    }
    recognition.start();
    """))

result = streamlit_bokeh_events(
    stt_button,
    events="GET_TEXT",
    key="listen",
    refresh_on_update=False,
    override_height=75,
    debounce_time=0)

if result:
    if "GET_TEXT" in result:
        st.write(result.get("GET_TEXT"))

This is pretty nice. Is it possible to add timer to this, so to record for 10 seconds for example and and save the text in a list? Sorry I am new to CustomJS(). Or is it possible to have a stop button also - so when the speech is done, one gets the text for output?

[BeyondMyself]

@ash2shukla

import streamlit as st
from bokeh.models.widgets import Button
from bokeh.models import CustomJS
from streamlit_bokeh_events import streamlit_bokeh_events

stt_button = Button(label="Speak", width=100)

stt_button.js_on_event("button_click", CustomJS(code="""
    var recognition = new webkitSpeechRecognition();
    recognition.continuous = true;
    recognition.interimResults = true;
 
    recognition.onresult = function (e) {
        var value = "";
        for (var i = e.resultIndex; i < e.results.length; ++i) {
            if (e.results[i].isFinal) {
                value += e.results[i][0].transcript;
            }
        }
        if ( value != "") {
            document.dispatchEvent(new CustomEvent("GET_TEXT", {detail: value}));
        }
    }
    recognition.start();
    """))

result = streamlit_bokeh_events(
    stt_button,
    events="GET_TEXT",
    key="listen",
    refresh_on_update=False,
    override_height=75,
    debounce_time=0)

if result:
    if "GET_TEXT" in result:
        st.write(result.get("GET_TEXT"))

This function supports English very good.
If I want to let it recognize other languages like Chinese, do you have some ideas?

[guozhiwei.8210]

@BeyondMyself You can set the recognition.lang = ‘cmn-Hans-CN’;

[guozhiwei.8210]

@ash2shukla Thanks for your idea and share the code, I have two questions to ask:

1. How to add punctuation marks based on the recognized content?
2. How to add a speech recognition end event to this button?

[saisowhit_P_B]

How to add a mike option in the text area?