Open Source Song Transcript and Video Caption Generator with Whisper

About:

I am working on building a product to serve clients on automating internal meeting summaries, song transcription for budding singers and product video review analysis in real time at an affordable cost.

Wanted to share a MVP I built in the initial stages for demonstration that would be useful for the community to build upon and target similar use cases in their respective areas of interest.

Features:

  • Grab any video from YouTube and generate Captions (which can be saved as SRT or VTT file) side by side of the video + segregated audio
  • Generate accurate lyrics for a song

To be updated:

  • Speaker Diarization
  • Word Level Captioning + Burning to Video

Demo Snapshot

Github Repo:

Click Here

I look forward to suggestions from the community :pray:. Any pointers towards streamlit-specific resources to host such GPU resource heavy applications on platforms such as VULTR (I have taken care of dockerization and tested with Heroku already) would be a great help!

1 Like

Interesting one. Probably more futureproof than downloading captions directely from youtube using youtube_transcript_api. It would be a nice feature to see how generated transcript differes from one downloaded directely :slight_smile:

As I’ve also was playing aroung with youtube videos summarization it would be nice to share experiences.

  1. Are you planning to use pre-trained summarization models using hugging face transformer library or openai API?

  2. How do you approach splitting text into chunks that models can handle?

1 Like

Dear @TomJohn,

Thanks for checking out the project.

    • The Whisper model generates better transcripts than the auto caption generator feature of YouTube due the fact that OpenAI seemed to have trained it (especially the largev2 and medium models) on data having different accents and incorporating multilingual audio of the same magnitude as well.
    • However, it completely ignores filler words like “Ah, umm…”. As far as transcripts/subtitles uploaded by YouTube user(s) themselves for the videos, for English, French, German and Japanese the results are quite similar.
    • For Indian languages like Tamil or Hindi, the generated transcripts are not as satisfactory
    • As far as songs are concerned especially rap and rock songs, it works incredibly well on most languages.
  1. Yes, my actual product involves video summarization + sentiment analysis so I am using pre-trained models from hugging face for those. Will build individual apis for those and integrate with streamlit

  2. The model(Whisper) can handle any video/audio for 1.5 hours easily. So generation of subtitles/transcripts is not an issue. As far as summarization is concerned with the generated transcript, the models based on Google T5 on huggingspace can handle even entire books so currently I did not encounter any requirement for splitting the text

1 Like

Thank you for detailed answer! :slight_smile:

1 Like