New Component: streamlit-webrtc, a new way to deal with real-time media streams

Custom Components streamlit-webrtc
, ,
62

@Joe_Tay

Hi,

Please paste the complete code that I can copy, paste and run to reproduce the problem.

Though I haven’t tested but just have seen it, let me point out some potential problems;

63

@whitphx

Hi,

Thank you for your reply and attention. I am trying to capture a series of high res images. Here is my code as requested:

Blockquote

import streamlit as st
import numpy as np
import pandas as pd
import threading
from typing import Union
import cv2
import av
import os
from streamlit_webrtc import VideoProcessorBase, webrtc_streamer

img_counter = 5

df = pd.DataFrame({
‘first column’: [‘Self Training’, ‘Self Analyse’],
})

sports option in side bar
s_option = st.sidebar.selectbox(
‘Select anlysis mode :’,
df[‘first column’])

def main():
class VideoTransformer(VideoProcessorBase):
frame_lock: threading.Lock # transform() is running in another thread, then a lock object is used here for thread-safety.
in_image: Union[np.ndarray, None]

    def __init__(self) -> None:
        self.frame_lock = threading.Lock()
        self.in_image = None
        self.img_list = []

    def recv(self, frame: av.VideoFrame) -> av.VideoFrame:
        in_image = frame.to_ndarray(format="bgr24")
        list_img = frame.to_ndarray(format="bgr24")

        global img_counter

        with self.frame_lock:
            self.in_image = in_image
            if img_counter > 0:
                #print("capturing image")
                self.img_list.append(list_img)
                img_counter -= 1

        return av.VideoFrame.from_ndarray(in_image, format="bgr24")

ctx = webrtc_streamer(key="snapshot", video_processor_factory=VideoTransformer)

if ctx.video_transformer:
    if st.button("Snapshot"):
        with ctx.video_transformer.frame_lock:
            in_image = ctx.video_transformer.in_image
            img_list = ctx.video_transformer.img_list
            

        if in_image is not None: #put in column form 5 images in a row
            st.write("Input image:")
            st.image(in_image, channels="BGR")
            st.image(img_list[0], channels="BGR") #lower resolution than the one above in_image
        else:
            st.warning("No frames available yet.")

if name == “main”:
main()

1 Like
64

Hi , I manage to solve the problem on my own. Thank you.

2 Likes
67

hey @whitphx thanks for such a useful library it was all i needed
i just made a small fun little streamlit app out of it

1 Like
68

@YashVardhan-AI Thank you, I’m glad to know that this component is useful for you.

69

Hi @whitphx , thank you again for your help. Just a quick question, would it be possible to change the resolution to HD at 1280, 720, it seems the default is 640, 480 for the frame.

recv(self, frame: av.VideoFrame) → av.VideoFrame:
in_image = frame.to_ndarray(format=“bgr24”)
print(str(in_image.shape))

Thank you

1 Like
70

@Joe_Tay Hi, I found that this is a problem having been discussed in the issue tracker of aiortc, which streamlit-webrtc depends on: How to change video stream size written by MediaRecorder? · Issue #183 · aiortc/aiortc · GitHub
I will investigate this and want to find a solution, but currently have no idea.

I created an issue, Unable to change the frame size · Issue #438 · whitphx/streamlit-webrtc · GitHub to track this.

New answer here: New Component: streamlit-webrtc, a new way to deal with real-time media streams - #74 by whitphx

1 Like
71

hey @whitphx welcome

1 Like
72

Hello, Good day.

I tried to set up with my Apple M1 device however, I have errors regarding with Failed to build aiortc av pylibsrtp

Hope someone can help me to understand this error (: Thank you

Screen Shot 2021-10-11 at 7.22.50 PM
Screen Shot 2021-10-11 at 7.22.50 PM1130×254 93.6 KB

1 Like
73

Thank you

1 Like
74

@Joe_Tay @nerilou.delacruz
My previous answer was wrong. Being unable to set the frame size is for the case of recorder, but not processor.

You can configure the input resolution as below:

webrtc_streamer(
    ...
    media_stream_constraints={
        "video": {
            "width": 1280,
        }
    },
    ...
)

or for finer controls:

webrtc_streamer(
    ...
    media_stream_constraints={
        "video": {
            "width": {"min": 800, "ideal": 1200, "max": 1920 },
        }
    },
    ...
)

media_stream_constraints parameter is passed to navigator.mediaDevices.getUserMedia() on the frontend code to configure the media devices.
This page, for example, explains the details of that API. If you want more controls, this page can be a hint.

Note that the media_stream_constraints setting is a instruction for the browser to configure the media devices, but the browser sometimes does not follow it.

75

@nerilou.delacruz It’s simply a problem of these libraries related to M1 chip, not this component, IMO.
These libs contain C code and have to be built at installation, so may have problems on M1 platform.

Just for a reference, I paste my recipe on M1 mac here restored from my memory and some documents.
pyenv installed via homebrew is required.

$ arch -arch x86_64 env PATH=${PATH/\/opt\/homebrew\/bin:/} pyenv install 3.8.7
$ pip install -U pip

The point is to install the python in an x86-emulated environment. arch command does it.
Setting the PATH envvar is to use the x86 version of the Python dependency libraries installed via homebrew, during pyenv install.
Using the latest version of pip may also be a point.

Note that I haven’t tested this instruction from top to bottom. There can be a missing part.
And I don’t support it. Please use this info just as a clue.

1 Like
76

Hi,
I just installed streamlit v.1.0 and want to use simply the webcam. So I tried the provided snippet from the newest blog entry:

from webcam_component import webcam

captured_image = webcam()
if captured_image is not None:
   st.image(captured_image)

It says: “ModuleNotFoundError: No module named ‘webcam_component’”. I installed your package with pip install streamlit-webrtc. It still does not work somehow…

1 Like
77

@Firehead1971
This is the blog post: Announcing Streamlit 1.0! 🎈 right?

Anyway, the sample code in it is not about this streamlit-webrtc component, but just shows an example of a usage of a third-party component.

Forwarding to @Adrien_Treuille who is probably the author of that blog post.

1 Like
78

Then it is a bit misleading because it looks like that a webcam can be used with just those lines of code. Anyway, I just used your repo :wink: Just another question, is it possible to access the streamlit session state from inside of the callback function of VideoProcessor?

79

is it possible to access the streamlit session state from inside of the callback function of VideoProcessor?

It’s impossible now.
However, it’s also an interesting point - I will investigate and think of the way to make it possible in the future versions.

1 Like
80

After some survey, I think it won’t be implemented because it seems to require not intuitive hacks to create.
However, as inspired from this conversation, I’m thinking of introducing a new state object like session_state shared both by inside and outside the callbacks.

For example, we will be able to use the shared state like below.

from streamlit_webrtc import shared_state

class VideoProcessor:
    def recv(self, frame):
        img = frame.to_ndarray(...)
        
        threshold = shared_state["threshold"]
        
        return av.VideoFrame.from_ndarray(img, ...)


webrtc_streamer(video_processor_factory=VideoProcessor)        

shared_state["threshold"] = st.slider("Threshold", ...)

Or, if such a shared_state is introduced, class-based video processors will no longer be needed and callables might be in that place as below.

from streamlit_webrtc import shared_state

def process_frame(frame):
    img = frame.to_ndarray(...)
    
    threshold = shared_state["threshold"]
    
    return av.VideoFrame.from_ndarray(img, ...)


webrtc_streamer(video_callback=process_frame)

shared_state["threshold"] = st.slider("Threshold", ...)

@Firehead1971
Do you think such design is helpful for your needs?
Can you tell me your use case where you want to pass the session_state directly to the callbacks?

And not only him but also everyone, please give me your opinion!

81

@ whitphx Thanks a lot for the great work, keep it up.

I was wondering if I can use the async def recv_queued function to provide a batch of frames to be processed by the model at once, i.e i will delay the stream by n secs, to be able to batch process it

is this applicable, and if not, what is your recommendation for providing the DL model a batch of frames at once instead of one frame at a time?

1 Like
82

@ahmedanis03 I think it’s possible though I haven’t actually tested it.
It would be a kind of extended versions of the delayed echo example.

For example,

  • Assume the input video is 10 FPS.
  • When recv_queued() is called first time, the input frames contain the first frame (probably len(frames) == 1).
    If you call await asyncio.sleep(1) inside recv_queued() and returns the frames from the callback, these frames are displayed with 1 sec delay.
  • When recv_queued() is called next time, the input frames contain frames that have arrived after the previous frames. Probably len(frames) == 10 because the previous recv_queued() took 1sec in which 10 frames should have arrived.
    So, now you can process these 10 frames at one time.
    If this batch processing takes 1 sec again, the same thing continues in the successive calls…
1 Like
83

@whitphx Just, maybe a simple question, when I am deploying the app on streamlit cloud it does not render the output of the webcam. Instead I get only a black screen. On localhost it works smoothly but on remote host it returns the black screen output.

MWE

import streamlit as st
import av
import cv2
from streamlit_webrtc import webrtc_streamer
from PIL import Image

webrtc_streamer(key="sample")

How can I fetch the video output when I am deploying the app on e.g. streamlit cloud?

1 Like