New Component: streamlit-webrtc, a new way to deal with real-time media streams

@Luxor5k
Post your full non-working code first to investigate the problem.

Without it, I only can suspect that you didn’t follow the change of the method signature from transform() to recv(). recv() has to return an instance of av.VideoFrame or av.AudioFrame, not np.ndarray.
See GitHub - whitphx/streamlit-webrtc: Real-time video and audio streams over the network, with Streamlit.

@AbdoShahat

the app runs in my localhost,
when I try to run your app, don’t run
and when I deploy my project in streamlit, it creates a public ssh key in my repo in Github

When I tried to make it from a function I’m getting this error

I suspect you manipulated st.session_state inside recv(), which is not supported (See GitHub - whitphx/streamlit-webrtc: Real-time video and audio streams over the network, with Streamlit.).

Sorry, but I can’t support you any more with these little informative and hard-to-read questions.
I suggest you to follow How do I ask a good question? - Help Center - Stack Overflow, especially the “Help others reproduce the problem” section.

1 Like

Hi everyone (and whitphx!)

First of all I am super amazed by your contribution to the streamlit community. I still haven’t come across a better alternative for manipulation video and classification! :partying_face:

I am currently using streamlit-webrtc for reading barcode and it totally works! :grin:

However as soon as I have detected a barcode I wish to have print the barcode like so st.subheader((“Detected barcode:” detected_barcode)), and then use barcode to fetch more information in a database etc. However I read that it’s hard to extract information from the webrtc_streamer object to global :smiling_face_with_tear:. Any ideas for a work around?

I could only come up with a complicated attempt, but I feel like it doesn’t need to be this complicated :sweat_smile::

  1. In the beginning of the script I would generate a random string
  2. Upon detecting a barcode, store this value in a new SQL entry WHERE id = random_string
  3. Then after initiating live_detection(), I would start a “while true” loop that would constantly look for new SQL entries WHERE id = random_string (this should work this it’s the same session). If found, then do st.subheader((“Detected barcode:” sql_barcode())) and del live_detection to freeze / collapse the videostream.
  4. At the same time I spawn a button that when triggered would reload the page/session and start live_detection() again.

Note: For barcodes, it is extremely painful to only use single images as the detection rate is super low and you often have to take 4-5 images before anything is detected. Video solves this problem beautifully.

Here is my current code (without attempting to write to a SQL database)

import av
import cv2
import streamlit as st 
from pyzbar.pyzbar import decode
from streamlit_webrtc import (webrtc_streamer, VideoProcessorBase,WebRtcMode)

def live_detection(play_state):

   class BarcodeProcessor(VideoProcessorBase):

      def __init__(self) -> None:
         self.barcode_val = False
      
      def BarcodeReader(self, image):
         detectedBarcodes = decode(image)
         if not detectedBarcodes:
            print("\n No barcode! \n")
            return image, False

         else:
            for barcode in detectedBarcodes: 
               (x, y, w, h) = barcode.rect
               cv2.rectangle(image, (x-10, y-10),
                              (x + w+10, y + h+10),
                              (0, 255, 0), 2)

            if detectedBarcodes[0] != "":
               return image, detectedBarcodes[0]


      def recv(self, frame: av.VideoFrame) -> av.VideoFrame:
         image = frame.to_ndarray(format="bgr24")

         annotated_image, result = self.BarcodeReader(image)
        
         if result == False:
            return av.VideoFrame.from_ndarray(image, format="bgr24")

         else:

            self.barcode_val = result[0]
            play_state = False
            return av.VideoFrame.from_ndarray(annotated_image, format="bgr24")

   stream = webrtc_streamer(
         key="barcode-detection",
         mode=WebRtcMode.SENDRECV,
         desired_playing_state=play_state,
         video_processor_factory=BarcodeProcessor,
         media_stream_constraints={"video": True, "audio": False},
         async_processing=True,
      )

play_state = True
detected_barcode = live_detection(play_state)

Okay my SQL was pretty bad. In other words, the while loop messed everything up haha

Update, I made a new solution work like ~25% but it’s very unstable and keeps crashing and introduces alot of lag. As you can see it correctly prints the barcode right after identification :slight_smile: , and I can use it to access a DB and grap more information.

image

Here is the new code:

import av
import cv2
import time
import streamlit as st 
from pyzbar.pyzbar import decode
from streamlit_webrtc import (webrtc_streamer, VideoProcessorBase,WebRtcMode)

st.set_page_config(layout="wide")

def live_detection(play_state):

   c1, c2 = st.columns(2)
   class BarcodeProcessor(VideoProcessorBase):

      def __init__(self) -> None:
         self.barcode_val = False
      
      def BarcodeReader(self, image):
         detectedBarcodes = decode(image)
         if not detectedBarcodes:
            print("\n No barcode! \n")
            return image, False

         else:
            for barcode in detectedBarcodes: 
               (x, y, w, h) = barcode.rect
               cv2.rectangle(image, (x-10, y-10),
                              (x + w+10, y + h+10),
                              (0, 255, 0), 2)

            if detectedBarcodes[0] != "":
               return image, detectedBarcodes[0]


      def recv(self, frame: av.VideoFrame) -> av.VideoFrame:
         image = frame.to_ndarray(format="bgr24")

         annotated_image, result = self.BarcodeReader(image)

         if result == False:
            return av.VideoFrame.from_ndarray(annotated_image, format="bgr24")
         else:
            self.barcode_val = result[0]
            return av.VideoFrame.from_ndarray(annotated_image, format="bgr24")

   stream = webrtc_streamer(
         key="barcode-detection",
         mode=WebRtcMode.SENDRECV,
         desired_playing_state=play_state,
         video_processor_factory=BarcodeProcessor,
         media_stream_constraints={"video": True, "audio": False},
         async_processing=True,
      )

   while True:
      if stream.video_processor.barcode_val != False:
         barcode = stream.video_processor.barcode_val
         print("FOUND")
         c1.subheader(barcode)
         del stream

play_state = True

possible_barcode = live_detection(play_state)

Hi @lorenzweb ,
Thank you for sharing your nice work!

As you came up with, polling with a while-loop and passing values via instance attributes are the only way to communicate from inside to outside the callback with the current API design of streamlit-webrtc.

Just to confirm; so now everything is ok?

Thanks for your answer!

It appears to work now :D. Seem like I needed to introduce a little delay (time.sleep(0.10) and it is able to update the barcode very smoothly :smiley: :partying_face:

As you can see the barcodes st.subheaders gets stacked. So I tried to use placeholder = st.empty() to clear out the old prints. but got the following error:

The final loop actually works quite well, only dealing with a few things.

I’m looking for a way to either a) collapse the streaming-window following barcode detection with the option of re-spawning the window after tapping a st.button(“Scan new barcode”) or b) continous updated of the last detected barcode with st.subheader() without stacking (see image above). By putting break in the while-loop (see code below), I can succesfully grap the detected barcode and use it for various actions - but this method doesn’t allow for automatic update. However the st.subheader is correctly updated if I hit a st.button() after scanny a new barcode :sweat_smile:

   while True:
      time.sleep(0.10)
      if (stream.video_processor.barcode_val != False) and stream.video_processor.barcode_val != old_barcode:
         old_barcode = stream.video_processor.barcode_val
         place_holder.empty()  ## THIS FAILS
         c1.subheader(old_barcode)
         stream.desired_playing_state= False ## THIS DOESN'T SEEM TO CLOSE THE STREAM
         break # THIS BREAK PREVENTS THE "STREAMLIT ICON" FROM STAYING IN RUNNING MODE, BUT THE VIDEO-STREAM STILL UDPATES THE LATEST BARCODE CORRECTLY, BUT DOESN*T PRINT WITH st.subheader() UNTIL A WIDGET IS PRESSED.

Thank you again so much for the amazing library. :partying_face: :partying_face: :partying_face:
Will share the final project if I can get everything working.

2 Likes

@lorenzweb Congrats! it’s fantastic :smile:

I have some comments;

  • As you wrote in the comment, setting stream.desired_playing_state does not work because this library does not support it. To affect the webrtc running state, we have to set the desired_playing_state argument on the webrtc_streamer() function.
    • So, if you want to programmatically stop the stream after the loop ends, you may have to call st.experimental_rerun() to execute the next run, where you will pass desired_playing_state=False to webrtc_streamer().
  • As you wrote in the comment, the break prevents the “RUNNING” indicator as it stops the while-loop, however, it stops the loop, of course, so that c1.subheader() is no more called and its content is no longer updated in the current run. So, if you want the content of the subheader to be updated continuously, you should not use break and keep the loop running.
    • For now, to fetch data from the webrtc worker and process it continuously in the app script, using loops is the only way. And in that case, we have to accept the “RUNNING” indicator to show during it.
1 Like

Hi, appreciate your work!

I’m new to Python and Streamlit. I’m trying to build a pose estimation project where I have to provide user with both textual and audio feedback as they perform an exercise. I have achieved the textual feedback part using VideoProcesser where I take the camera feed frame by frame and with the help of trained mediapipe model display the name of performed pose to user. Now I have to also output the audio for this. I cannot figure out how to add audio feedback along with video frame processing.

Is this even possible with this technology to give audio feedback along with processing video frames?

Thank you so much.

Can I access the Start/Stop button? I want to perform some action upon button click.

When you initialize your stream object, you can enable audio like this:

image

Dear Whitphx,

Thanks for the cool library!

I managed to run the webrtc at home without any issue. I then tried to show my colleagues at work using a company PC) but it just started freezing/hanging after I hit Start (note: I am able to choose webcam preview in select device)

image

After hitting start I noticed that the ICE connection refuse to go to “ICE connection state is completed” :smiling_face_with_tear:

Instead it closes and throws this message:

I feel like these issues could relate to perhaps IT / network security we have, but I’m a bit stuck debugging. What is your best bet?

@Hafsah_MR

audio feedback

It may be achieved by using AudioProcessor.

In the official example, currently there are only samples that transform input audio into output, however, overriding the audio inside AudioProcessor.recv() also seems possible - I haven’t tried and am not sure though.

One drawback of this approach is that it requires the input audio even though it is not used. This is a limitation of the current API, as it only supports custom filters, but not custom sources.

Can I access the Start/Stop button?

The event handler on button click is not supported now (it’s under development now: feature/callback by whitphx · Pull Request #695 · whitphx/streamlit-webrtc · GitHub . Please wait for the release.)

Instead, you can refer to ctx.state.playing for status checking.

@ZKLO
That error is being tracked in Connection is shutdown and errors appear in some network environment · Issue #552 · whitphx/streamlit-webrtc · GitHub, but has not resolved yet.

I think, however, this error message itself is a subsequent event following a network trouble, and the fundamental problem may resides in the network architecture as this comment. For example, there are firewalls that drop WebRTC packets. This often occurs in office or mobile networks.

So GitHub - whitphx/streamlit-webrtc: Real-time video and audio streams over the network, with Streamlit. would be the help.

This post seems to be related too.

I released streamlit-webrtc v0.37.0 :v:

This new version supports

3 Likes

Continuous amazing work @whitphx!! :balloon:

2 Likes

@whitphx When deloy object detection by webrtc streamer, how do I show fps at other column? As I understood, can’t run st.markdown at recv().

col_monitor, col_result = st.columns((6, 2)) # webcam here
        col_cam, col_rep_img, col_detail = st.columns((6, 1, 1))
        col_fr, col_dt, col_iw, _  = st.columns((2, 2, 2, 2))
        kpi1_text = None
        with col_fr:
              st.markdown('**Frame Rate of Camera**') # I want to show fps here
             kpi1_text = st.markdown("0")