New Component: streamlit-webrtc, a new way to deal with real-time media streams

@Firehead1971
I think it’s because WebRTC connection cannot be established in your network environment due to for example network topology, firewall, etc.
This issue comment might be a solution.

BTW, this is a frequently asked question, so I’m planning to write a document about it.GitHub - whitphx/streamlit-webrtc: Real-time video and audio streams over the network, with Streamlit.

Hi,

I published a new tutorial for real-time video app development using streamlit-webrtc (on my new blog :slight_smile: ):

( Crosspost to dev.to: Developing web-based real-time video/audio processing apps quickly with Streamlit - DEV Community )

1 Like

It’s now working, thank youuuu :clap: :raised_hands:

Hi everyone I need to mute and unmute the audio of the video streaming after making an action.
Any suggestions on how can I do this ?

1 Like

@quiver99
How about this?

import streamlit as st
from streamlit_webrtc import webrtc_streamer, VideoHTMLAttributes

muted = st.checkbox("Mute")

webrtc_streamer(
    key="mute_sample",
    video_html_attrs=VideoHTMLAttributes(
        autoPlay=True, controls=True, style={"width": "100%"}, muted=muted
    ),
)

You can configure the <video> element’s attribute via the video_html_attrs argument,
and the muted attribute switches mute/unmute on the client-side.

Note that other attributes such as autoPlay and style are specified too in the example above.
This default set of the video attribute values is used when nothing is provided for the video_html_attrs argument,
but if you override it, you have to explicitly provide these necessary attributes.

Thank for the reply @whitphx!
Any chance to do it without using checkboxs
Like calling it from a function or something like that
Thanks in advance

1 Like

Hi everyone is there any chance to change the name of the start, stop and select device buttons ? I’m making an app in spanish and I need to translate the names and I’ve tried to change the name in the js file in the route \Lib\site-packages\streamlit_webrtc\frontend\build\static\js\main.2d1be7e6.js but when I deploy it to heroku the changes doesn’t save
How can i change it ?

Hi! I have a problem, when I try to exec this code the console shows transform() is deprecated. I implement recv, but when I change it my code doesn’t work anymore

class VideoTransformer(VideoTransformerBase):
    @staticmethod
    def transform_(img: np.array) -> np.array:
        face_to_predict, list_ubications = prepara_imagen_array(img=img)
        list_clases = get_predictions(face_to_predict=face_to_predict)

        if len(list_clases) > 0:

            for enum in range(len(list_clases)):
                x, y, w, h = list_ubications[enum]
                img = cv2.rectangle(img, (x, y), (x + w, y + h), color_dict[list_clases[enum]], 2)
                img = cv2.rectangle(img, (x, y - 40), (x+w, y), color_dict[list_clases[enum]], -2)
                img = cv2.putText(img, labels_dict[list_clases[enum]], (x, y - 10), cv2.FONT_HERSHEY_COMPLEX, 0.75,
                                  (255, 255, 255), 1, cv2.LINE_AA)
        return img, list_clases

    def transform(self, frame):
        img = frame.to_ndarray(format="bgr24")
        img, list_clases = VideoTransformer.transform_(img=img)
        return img

I tried to detect facial mask in persons like this

@whitphx any solution?

Hello, any help
When I try to start the camera on your demo site, it doesn’t load. My camera seems to be on but the camera is just loading and then resetting. and when I try whitphx website did the same

1 Like

@quiver99
Using a checkbox is just an example, and the essential part is controlling the muted state via the variable.
So you should apply it to your code and use-case.

If you want to control it from a function, mutate the variable from it.
The session state may be helpful for that purpose.

import streamlit as st

from streamlit_webrtc import VideoHTMLAttributes, webrtc_streamer

if "muted" not in st.session_state:
    st.session_state["muted"] = False


def callback():
    st.session_state["muted"] = not st.session_state["muted"]


st.button("Toggle mute", on_click=callback)


muted = st.session_state["muted"]

webrtc_streamer(
    key="mute_sample",
    video_html_attrs=VideoHTMLAttributes(
        autoPlay=True, controls=True, style={"width": "100%"}, muted=muted
    ),
)

@quiver99 Changing the button text is not supported now.

→ I created a ticket Make the button texts editable · Issue #730 · whitphx/streamlit-webrtc · GitHub . Please track it.

@Luxor5k
Post your full non-working code first to investigate the problem.

Without it, I only can suspect that you didn’t follow the change of the method signature from transform() to recv(). recv() has to return an instance of av.VideoFrame or av.AudioFrame, not np.ndarray.
See GitHub - whitphx/streamlit-webrtc: Real-time video and audio streams over the network, with Streamlit.

@AbdoShahat

the app runs in my localhost,
when I try to run your app, don’t run
and when I deploy my project in streamlit, it creates a public ssh key in my repo in Github

1 Like

When I tried to make it from a function I’m getting this error

I suspect you manipulated st.session_state inside recv(), which is not supported (See GitHub - whitphx/streamlit-webrtc: Real-time video and audio streams over the network, with Streamlit.).

Sorry, but I can’t support you any more with these little informative and hard-to-read questions.
I suggest you to follow How do I ask a good question? - Help Center - Stack Overflow, especially the “Help others reproduce the problem” section.

1 Like

Hi everyone (and whitphx!)

First of all I am super amazed by your contribution to the streamlit community. I still haven’t come across a better alternative for manipulation video and classification! :partying_face:

I am currently using streamlit-webrtc for reading barcode and it totally works! :grin:

However as soon as I have detected a barcode I wish to have print the barcode like so st.subheader((“Detected barcode:” detected_barcode)), and then use barcode to fetch more information in a database etc. However I read that it’s hard to extract information from the webrtc_streamer object to global :smiling_face_with_tear:. Any ideas for a work around?

I could only come up with a complicated attempt, but I feel like it doesn’t need to be this complicated :sweat_smile::

  1. In the beginning of the script I would generate a random string
  2. Upon detecting a barcode, store this value in a new SQL entry WHERE id = random_string
  3. Then after initiating live_detection(), I would start a “while true” loop that would constantly look for new SQL entries WHERE id = random_string (this should work this it’s the same session). If found, then do st.subheader((“Detected barcode:” sql_barcode())) and del live_detection to freeze / collapse the videostream.
  4. At the same time I spawn a button that when triggered would reload the page/session and start live_detection() again.

Note: For barcodes, it is extremely painful to only use single images as the detection rate is super low and you often have to take 4-5 images before anything is detected. Video solves this problem beautifully.

Here is my current code (without attempting to write to a SQL database)

import av
import cv2
import streamlit as st 
from pyzbar.pyzbar import decode
from streamlit_webrtc import (webrtc_streamer, VideoProcessorBase,WebRtcMode)

def live_detection(play_state):

   class BarcodeProcessor(VideoProcessorBase):

      def __init__(self) -> None:
         self.barcode_val = False
      
      def BarcodeReader(self, image):
         detectedBarcodes = decode(image)
         if not detectedBarcodes:
            print("\n No barcode! \n")
            return image, False

         else:
            for barcode in detectedBarcodes: 
               (x, y, w, h) = barcode.rect
               cv2.rectangle(image, (x-10, y-10),
                              (x + w+10, y + h+10),
                              (0, 255, 0), 2)

            if detectedBarcodes[0] != "":
               return image, detectedBarcodes[0]


      def recv(self, frame: av.VideoFrame) -> av.VideoFrame:
         image = frame.to_ndarray(format="bgr24")

         annotated_image, result = self.BarcodeReader(image)
        
         if result == False:
            return av.VideoFrame.from_ndarray(image, format="bgr24")

         else:

            self.barcode_val = result[0]
            play_state = False
            return av.VideoFrame.from_ndarray(annotated_image, format="bgr24")

   stream = webrtc_streamer(
         key="barcode-detection",
         mode=WebRtcMode.SENDRECV,
         desired_playing_state=play_state,
         video_processor_factory=BarcodeProcessor,
         media_stream_constraints={"video": True, "audio": False},
         async_processing=True,
      )

play_state = True
detected_barcode = live_detection(play_state)

1 Like

Okay my SQL was pretty bad. In other words, the while loop messed everything up haha

1 Like

Update, I made a new solution work like ~25% but it’s very unstable and keeps crashing and introduces alot of lag. As you can see it correctly prints the barcode right after identification :slight_smile: , and I can use it to access a DB and grap more information.

image

Here is the new code:

import av
import cv2
import time
import streamlit as st 
from pyzbar.pyzbar import decode
from streamlit_webrtc import (webrtc_streamer, VideoProcessorBase,WebRtcMode)

st.set_page_config(layout="wide")

def live_detection(play_state):

   c1, c2 = st.columns(2)
   class BarcodeProcessor(VideoProcessorBase):

      def __init__(self) -> None:
         self.barcode_val = False
      
      def BarcodeReader(self, image):
         detectedBarcodes = decode(image)
         if not detectedBarcodes:
            print("\n No barcode! \n")
            return image, False

         else:
            for barcode in detectedBarcodes: 
               (x, y, w, h) = barcode.rect
               cv2.rectangle(image, (x-10, y-10),
                              (x + w+10, y + h+10),
                              (0, 255, 0), 2)

            if detectedBarcodes[0] != "":
               return image, detectedBarcodes[0]


      def recv(self, frame: av.VideoFrame) -> av.VideoFrame:
         image = frame.to_ndarray(format="bgr24")

         annotated_image, result = self.BarcodeReader(image)

         if result == False:
            return av.VideoFrame.from_ndarray(annotated_image, format="bgr24")
         else:
            self.barcode_val = result[0]
            return av.VideoFrame.from_ndarray(annotated_image, format="bgr24")

   stream = webrtc_streamer(
         key="barcode-detection",
         mode=WebRtcMode.SENDRECV,
         desired_playing_state=play_state,
         video_processor_factory=BarcodeProcessor,
         media_stream_constraints={"video": True, "audio": False},
         async_processing=True,
      )

   while True:
      if stream.video_processor.barcode_val != False:
         barcode = stream.video_processor.barcode_val
         print("FOUND")
         c1.subheader(barcode)
         del stream

play_state = True

possible_barcode = live_detection(play_state)
1 Like

Hi @lorenzweb ,
Thank you for sharing your nice work!

As you came up with, polling with a while-loop and passing values via instance attributes are the only way to communicate from inside to outside the callback with the current API design of streamlit-webrtc.

Just to confirm; so now everything is ok?