Processing video with OpenCV and write it to disk (to display in st.video)

[zharsky]

Hello everyone! So great to use Streamlit for my projects, thank you so much for your work and dedication!

My question is something similar to this topic. I also want to upload video to my Streamlit app. And I also want to process each frame with OpenCV functions (and also with some Neural Network model).

What I want to do next is to display a processed video on my web page generated with Streamlit app. In the topic that I attached to this question before, the author just displays processed frames one by one with st.empty() method. But this will not work for me. As I mentioned I am going to use some heavy functions and NN models to process my video. So processing of the full 5 sec video row might take around 50 sec. If I display each frame of a video after it’s full processing, the frame rate will be just too low. So instead I want first to fully process the video and only then I want to display it on my web app, ideally with st.video() method. The best way to do that in my opinion is to write a ready video on a drive using OpenCV method cv2.VideoWriter and then open it as BytesIO object and display it with st.video() method.

So as I see a full pipeline, I want to upload a video to my app, save it as a temporary file on a drive at my backend side, read it with OpenCV (cv2.VideoCapture), process frame by frame and save it frame by frame with cv2.VideoWriter method to another temporary file and then open it and run with st.video() method.

My problem is that I can’t write a video to a drive with cv2.VideoWriter. This method works well when I run it from console or IDE, but it doesn’t work when I run my Streamlit app (it creates temp file on my drive, but its size is zero). I just don’t understand what is wrong here. Here’s the code:

import streamlit as st
import cv2

video_data = st.file_uploader("Upload file", ['mp4','mov', 'avi'])

# func to save BytesIO on a drive
def write_bytesio_to_file(filename, bytesio):
    """
    Write the contents of the given BytesIO to a file.
    Creates the file or overwrites the file if it does
    not exist yet. 
    """
    with open(filename, "wb") as outfile:
        # Copy the BytesIO stream to the output file
        outfile.write(bytesio.getbuffer())

if video_data:
    # save uploaded video to disc
    temp_file_to_save = 'c:/temp_file_1.mp4'
    write_bytesio_to_file(temp_file_to_save, video_data)
    # read it with cv2.VideoCapture(), so now we can process it with OpenCV functions
    cap = cv2.VideoCapture(temp_file_to_save)
    # grab some parameters of video to use them for writing a new, processed video
    width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
    frame_fps = int(cap.get(cv2.CAP_PROP_FPS))
    st.write(width, height, frame_fps)
    # specify a writer to write a processed video to a disk frame by frame
    temp_file_result = 'c:/temp_file_2.mp4'
    fourcc_mp4 = cv2.VideoWriter_fourcc(*'XVID')
    out_mp4 = cv2.VideoWriter(temp_file_result, fourcc_mp4, frame_fps, (width, height))
    # loop though a video, process each frame and save it to a disk
    while True:
        ret, frame = cap.read()
        # if frame is read correctly ret is True
        if not ret:
            st.write("Can't receive frame (stream end?). Exiting ...")
            break
        # some video processing here
        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        # write a processed frame to the video on a disk
        out_mp4.write(gray)

    # when video is fully saved to disk, open it as BytesIO and play with st.video()
    result_video = open(temp_file_result, "rb")
    st.video(result_video)

Any ideas why it doesn’t work? Maybe some conflict between a loop of Streamlit (as it reads through the whole code again and again) and a loop that processes and saves a video? Maybe there is some other way to implement my plan? I will be glad to get any help. I am also open for any ideas. Thank you!

[edsaac]

TLDR: you need to get the cv2.VideoWriter_fourcc(*'h264') codec to work if you want your mp4 video compatible with streamlit’s web player.

This was a fun puzzle!. Turns out it’s more of an OpenCV issue, rather than a streamlit problem but here it goes.

• To generate an mp4 file you should use an appropriate codec, such as “mp4v” or “h264”. In your case, it is probably falling back to mp4v as “xvid” expects a .avi file.
• The MIME type error comes from the “mp4v” codec as MPEG-4 is not supported by streamlit’s html5 web player. You would need to set the “h264” (or “x264”) codec to make the video compatible, however, that support seems to not come by default with OpenCV (see issue).

Here is a little demo using ffmpeg to reencode to H264 the video written by OpenCV:

The code:

import streamlit as st
import cv2
import subprocess

video_data = st.file_uploader("Upload file", ['mp4','mov', 'avi'])

temp_file_to_save = './temp_file_1.mp4'
temp_file_result  = './temp_file_2.mp4'

# func to save BytesIO on a drive
def write_bytesio_to_file(filename, bytesio):
    """
    Write the contents of the given BytesIO to a file.
    Creates the file or overwrites the file if it does
    not exist yet. 
    """
    with open(filename, "wb") as outfile:
        # Copy the BytesIO stream to the output file
        outfile.write(bytesio.getbuffer())

if video_data:
    # save uploaded video to disc
    write_bytesio_to_file(temp_file_to_save, video_data)

    # read it with cv2.VideoCapture(), 
    # so now we can process it with OpenCV functions
    cap = cv2.VideoCapture(temp_file_to_save)

    # grab some parameters of video to use them for writing a new, processed video
    width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
    frame_fps = cap.get(cv2.CAP_PROP_FPS)  ##<< No need for an int
    st.write(width, height, frame_fps)
    
    # specify a writer to write a processed video to a disk frame by frame
    fourcc_mp4 = cv2.VideoWriter_fourcc(*'mp4v')
    out_mp4 = cv2.VideoWriter(temp_file_result, fourcc_mp4, frame_fps, (width, height),isColor = False)
   
    while True:
        ret,frame = cap.read()
        if not ret: break
        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) ##<< Generates a grayscale (thus only one 2d-array)
        out_mp4.write(gray)
    
    ## Close video files
    out_mp4.release()
    cap.release()

    ## Reencodes video to H264 using ffmpeg
    ##  It calls ffmpeg back in a terminal so it fill fail without ffmpeg installed
    ##  ... and will probably fail in streamlit cloud
    convertedVideo = "./testh264.mp4"
    subprocess.call(args=f"ffmpeg -y -i {temp_file_result} -c:v libx264 {convertedVideo}".split(" "))
    
    ## Show results
    col1,col2 = st.columns(2)
    col1.header("Original Video")
    col1.video(temp_file_to_save)
    col2.header("Output from OpenCV (MPEG-4)")
    col2.video(temp_file_result)
    col2.header("After conversion to H264")
    col2.video(convertedVideo)

PDs:

• Don’t forget to close the video files you are writing to and reading from. The files should be closed automatically after the script terminates, but in streamlit the file you are writing to would never released. Just add cap.release() and out_mp4.release() after finish reading and writing.
• When writing a grayscale video, don’t forget to add that flag (isColor=False) to the VideoWriter.
• It’s better to use relative file paths like ./myvideo.mp4, rather than the windows-only C:/myvideo.mp4

edit:

• Instead of running ffmpeg as a subprocess, you could try the ffmpy package to reencode the video .

[zharsky]

Hey Edwin! This is great! Thank you so much for the detailed and comprehensive answer! You’ve made a great analyze and pointed out few errors I’ve made. Thank you so much for your time!

Just to conclude, the real problem here was the fact that I didn’t close the video files after their processing with OpenCV. Just like you mentioned I expected them to be closed automatically when the script terminates (in fact I didn’t even think of that and didn’t expect anything, because it always just closes automatically when I run a script) but in Streamlit loop it doesn’t work like this, so files didn’t close and video wasn’t saved to drive (which was a specified problem of this issue). So the line of code that solves my problem is out_mp4.release().

But you went even further and specified a problem with h264 codec. Indeed I can’t open a video file encoded with mp4v in web player and it’s true that I can’t just encode the video in h264 with the vanilla version of OpenCV. So I will need to compile it manually or use ffmpy library to reencode the video (this option I maybe like better…)

Also I am wondering which way will work at Streamlit Share platform as long as eventually I want to deploy and publish my application there… Something tells me that for this scenario manual compiling of OpenCV is not a workable case. If you have any ideas or advices here I will be very grateful to you for them. If deploying and rendering to h264 at Streamlit Share become a real problem I will create a new topic for it.

Thank you for your help again!!

[Paramjit_Singh]

Hi @edsaac
I have written “opencv-python-headless” in requirements.txt
Still getting this error:
ImportError: This app has encountered an error. The original error message is redacted to prevent data leaks. Full error details have been recorded in the logs (if you’re on Streamlit Cloud, click on ‘Manage app’ in the lower right of your app).

Traceback:

File "/home/appuser/venv/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 565, in _run_script
    exec(code, module.__dict__)File "/app/new/pages/2-Video_Dehazer.py", line 2, in <module>
    import cv2

[leonidasavila]

Hi guys! This is my first post. I’m new in streamlit and I’m building an app with opencv and mediapipe.

I tried the code of @edsaac but i got this error. Any idea that what’s going on? Thank,