New Component: streamlit-webrtc, a new way to deal with real-time media streams

is it possible to access the streamlit session state from inside of the callback function of VideoProcessor?

It’s impossible now.
However, it’s also an interesting point - I will investigate and think of the way to make it possible in the future versions.

1 Like

After some survey, I think it won’t be implemented because it seems to require not intuitive hacks to create.
However, as inspired from this conversation, I’m thinking of introducing a new state object like session_state shared both by inside and outside the callbacks.

For example, we will be able to use the shared state like below.

from streamlit_webrtc import shared_state

class VideoProcessor:
    def recv(self, frame):
        img = frame.to_ndarray(...)
        
        threshold = shared_state["threshold"]
        
        return av.VideoFrame.from_ndarray(img, ...)


webrtc_streamer(video_processor_factory=VideoProcessor)        

shared_state["threshold"] = st.slider("Threshold", ...)

Or, if such a shared_state is introduced, class-based video processors will no longer be needed and callables might be in that place as below.

from streamlit_webrtc import shared_state

def process_frame(frame):
    img = frame.to_ndarray(...)
    
    threshold = shared_state["threshold"]
    
    return av.VideoFrame.from_ndarray(img, ...)


webrtc_streamer(video_callback=process_frame)

shared_state["threshold"] = st.slider("Threshold", ...)

@Firehead1971
Do you think such design is helpful for your needs?
Can you tell me your use case where you want to pass the session_state directly to the callbacks?

And not only him but also everyone, please give me your opinion!

@ whitphx Thanks a lot for the great work, keep it up.

I was wondering if I can use the async def recv_queued function to provide a batch of frames to be processed by the model at once, i.e i will delay the stream by n secs, to be able to batch process it

is this applicable, and if not, what is your recommendation for providing the DL model a batch of frames at once instead of one frame at a time?

@ahmedanis03 I think it’s possible though I haven’t actually tested it.
It would be a kind of extended versions of the delayed echo example.

For example,

  • Assume the input video is 10 FPS.
  • When recv_queued() is called first time, the input frames contain the first frame (probably len(frames) == 1).
    If you call await asyncio.sleep(1) inside recv_queued() and returns the frames from the callback, these frames are displayed with 1 sec delay.
  • When recv_queued() is called next time, the input frames contain frames that have arrived after the previous frames. Probably len(frames) == 10 because the previous recv_queued() took 1sec in which 10 frames should have arrived.
    So, now you can process these 10 frames at one time.
    If this batch processing takes 1 sec again, the same thing continues in the successive calls…
1 Like

@whitphx Just, maybe a simple question, when I am deploying the app on streamlit cloud it does not render the output of the webcam. Instead I get only a black screen. On localhost it works smoothly but on remote host it returns the black screen output.

MWE

import streamlit as st
import av
import cv2
from streamlit_webrtc import webrtc_streamer
from PIL import Image

webrtc_streamer(key="sample")

How can I fetch the video output when I am deploying the app on e.g. streamlit cloud?

@Firehead1971
I think it’s because WebRTC connection cannot be established in your network environment due to for example network topology, firewall, etc.
This issue comment might be a solution.

BTW, this is a frequently asked question, so I’m planning to write a document about it.GitHub - whitphx/streamlit-webrtc: Real-time video and audio streams over the network, with Streamlit.

Hi,

I published a new tutorial for real-time video app development using streamlit-webrtc (on my new blog :slight_smile: ):

( Crosspost to dev.to: Developing web-based real-time video/audio processing apps quickly with Streamlit - DEV Community )

1 Like

It’s now working, thank youuuu :clap: :raised_hands:

Hi everyone I need to mute and unmute the audio of the video streaming after making an action.
Any suggestions on how can I do this ?

@quiver99
How about this?

import streamlit as st
from streamlit_webrtc import webrtc_streamer, VideoHTMLAttributes

muted = st.checkbox("Mute")

webrtc_streamer(
    key="mute_sample",
    video_html_attrs=VideoHTMLAttributes(
        autoPlay=True, controls=True, style={"width": "100%"}, muted=muted
    ),
)

You can configure the <video> element’s attribute via the video_html_attrs argument,
and the muted attribute switches mute/unmute on the client-side.

Note that other attributes such as autoPlay and style are specified too in the example above.
This default set of the video attribute values is used when nothing is provided for the video_html_attrs argument,
but if you override it, you have to explicitly provide these necessary attributes.

Thank for the reply @whitphx!
Any chance to do it without using checkboxs
Like calling it from a function or something like that
Thanks in advance

Hi everyone is there any chance to change the name of the start, stop and select device buttons ? I’m making an app in spanish and I need to translate the names and I’ve tried to change the name in the js file in the route \Lib\site-packages\streamlit_webrtc\frontend\build\static\js\main.2d1be7e6.js but when I deploy it to heroku the changes doesn’t save
How can i change it ?

Hi! I have a problem, when I try to exec this code the console shows transform() is deprecated. I implement recv, but when I change it my code doesn’t work anymore

class VideoTransformer(VideoTransformerBase):
    @staticmethod
    def transform_(img: np.array) -> np.array:
        face_to_predict, list_ubications = prepara_imagen_array(img=img)
        list_clases = get_predictions(face_to_predict=face_to_predict)

        if len(list_clases) > 0:

            for enum in range(len(list_clases)):
                x, y, w, h = list_ubications[enum]
                img = cv2.rectangle(img, (x, y), (x + w, y + h), color_dict[list_clases[enum]], 2)
                img = cv2.rectangle(img, (x, y - 40), (x+w, y), color_dict[list_clases[enum]], -2)
                img = cv2.putText(img, labels_dict[list_clases[enum]], (x, y - 10), cv2.FONT_HERSHEY_COMPLEX, 0.75,
                                  (255, 255, 255), 1, cv2.LINE_AA)
        return img, list_clases

    def transform(self, frame):
        img = frame.to_ndarray(format="bgr24")
        img, list_clases = VideoTransformer.transform_(img=img)
        return img

I tried to detect facial mask in persons like this

@whitphx any solution?

Hello, any help
When I try to start the camera on your demo site, it doesn’t load. My camera seems to be on but the camera is just loading and then resetting. and when I try whitphx website did the same

@quiver99
Using a checkbox is just an example, and the essential part is controlling the muted state via the variable.
So you should apply it to your code and use-case.

If you want to control it from a function, mutate the variable from it.
The session state may be helpful for that purpose.

import streamlit as st

from streamlit_webrtc import VideoHTMLAttributes, webrtc_streamer

if "muted" not in st.session_state:
    st.session_state["muted"] = False


def callback():
    st.session_state["muted"] = not st.session_state["muted"]


st.button("Toggle mute", on_click=callback)


muted = st.session_state["muted"]

webrtc_streamer(
    key="mute_sample",
    video_html_attrs=VideoHTMLAttributes(
        autoPlay=True, controls=True, style={"width": "100%"}, muted=muted
    ),
)

@quiver99 Changing the button text is not supported now.

→ I created a ticket Make the button texts editable · Issue #730 · whitphx/streamlit-webrtc · GitHub . Please track it.

@Luxor5k
Post your full non-working code first to investigate the problem.

Without it, I only can suspect that you didn’t follow the change of the method signature from transform() to recv(). recv() has to return an instance of av.VideoFrame or av.AudioFrame, not np.ndarray.
See GitHub - whitphx/streamlit-webrtc: Real-time video and audio streams over the network, with Streamlit.

@AbdoShahat

the app runs in my localhost,
when I try to run your app, don’t run
and when I deploy my project in streamlit, it creates a public ssh key in my repo in Github

When I tried to make it from a function I’m getting this error