STqdm : a tqdm-like progress bar for streamlit

Show the Community!
#1

Hello there !

I integrated tqdm with streamlit.
As discussed here, I am sharing the project without the community. :slight_smile:

Link to :

Install :
pip install stqdm

Some examples :

  • Directly
from time import sleep

from stqdm import stqdm

for _ in stqdm(range(50)):
    for _ in stqdm(range(15)):
        sleep(0.5)

nested_stqdm
nested_stqdm1249×393 703 KB

  • Using pandas
from time import sleep

import pandas as pd
from stqdm import stqdm

stqdm.pandas()

pd.Series(range(50)).progress_map(lambda x: sleep(1))
pd.Dataframe({"a": range(50)}).progress_apply(lambda x: sleep(1), axis=1)
13 Likes
🗞️ Weekly Roundup: New website, creating GitHub READMEs, GOT analysis and more!
Streamlit Components - Community Tracker
#2

Awesome! @adrien always wanted this :smiley:

#3

OMG THIS IS SO AWESOME!! Looking forward to using this in every project @Wirg!

1 Like
#4

@Wirg eheh :slight_smile: nice seeing you here!

Could you add your “component” to the Component tracker so we keep “track” of it :stuck_out_tongue: ?

Cheers,
Fanilo

#5

Hi :slight_smile:

I hope this will prove useful.

@andfanilo I think I did it. I updated the post. Is this what you meant ?

Have a nice day

#6

Yes this is perfect :slight_smile: thanks a lot for your work!

#7

Really awesome!!

#8

This is awesome!

I think it could be included natively in streamlit. What do you all think?

#9

Hi @luca,

Thank you for the suggestion.
I think the standard way to go about it will be to include it in tqdm itself.
tqdm is already doing this kind of switch to work seamlessly whether you are working in a normal python script, a notebook, or something else.
This is also going in streamlit direction to be usable both seamlessly in a normal python script or in a streamlit app.

I think that before doing that kind of move, the package has to prove stable, usable, and used :

  • I guess that tqdm or streamlit team has no interest in paying the toll of maintaining something that will be used by only 10 people
  • on the other hand, the package itself is not quite mature yet, mainly because the only feedback I have is mine. :wink: Integrating it into a bigger library could slow up development. Right now I am thinking of some improvements: using a single widget (rather than 2) by integrating the text into the progress bar, adding colors (at least to signify errors and help debug a user find the failing iteration).

I don’t know if other people agree with me on this?

  • waiting to integrate the package to a bigger one
  • the next improvements
2 Likes
#10

Hi !

Quick check for everyone using stqdm ! :slight_smile: I am happy to have you onboard. :wink:

What are you feeling on the package ?
Is it useful ?
Do you feel like missing some features ?

Have a nice day !

2 Likes
#11

Hey, I really like using it so far. I am having a little bit of trouble however when trying to use it to display upload progress when uploading to S3. The tqdm progress bar in the console prints fine, but I can’t seem to get the progress bar to display in my Streamlit app. Do you have any tips on how I could diagnose this behavior?

import streamlit as st
import boto3
from botocore.client import Config
from stqdm import stqdm

BUCKET_FOLDER = 'my_bucket'
KB = float(1024)

def hook(t):
    """
    params:
        t (tqdm): Accepts tqdm class
    """
    def inner(bytes_amount):
        bytes_amount = bytes_amount / KB
        t.update(bytes_amount)

    return inner


def upload_file_to_s3(upload_obj):
    """
    params:
        upload_obj (UploadedFile): Accepts Streamlit's uploaded file class
    """
    s3 = boto3.client('s3', region_name='us-east-2', config=Config(s3={'addressing_style': 'path'}), )
    upload_location = BUCKET_FOLDER + upload_obj.name
    xlsx_bytes = io.BytesIO(upload_obj.getvalue())

    try:
        with stqdm(total=upload_obj.size / KB,
                   unit='kB', 
                   unit_scale=True,
                   unit_divisor=KB,
                   desc=upload_obj.name,
                   leave=True) as t:
            s3.upload_fileobj(Fileobj=xlsx_bytes, Bucket=BUCKET_NAME, Key=upload_location, Callback=hook(t))

    except Exception as e:
        st.error(e)

My console outputs:

my_file.xlsx: 100%|██████████| 11.7/11.7 [00:00<00:00, 15.1kB/s]2021-04-29 13:42:05.476 Thread 'ThreadPoolExecutor-1_0': missing ReportContext

But the streamlit application’s stqdm bar looks like:

my_file.xlsx: 0% 0.00/11.7 [00:00<?, ?kB/s]
|□□□□□□□□□□|
#12

Hi @mihir,

Thanks for asking, I have been trying to reproduce and i think I am lacking some context. :slight_smile:

Is the working console output coming from the console output of the server with streamlit run ?
What is happening on streamlit side ? Is the progress bar stuck ?

From what I have tried, download / upload would not work at all. The reason for that is that upload_fileobj & download_fileobj are creating threads.
From what I know of streamlit sofar, subthreading can produce issues if not handled properly.
For example, I have a missing ReportContext when running a slightly modified version of your code. Improve "missing ReportContext" threading error · Issue #1326 · streamlit/streamlit · GitHub

I did not find a way to force that in boto3, but you can ask boto3’s fileobj functions not to produce thread by adding it to the config.
Using s3.download_fileobj(Bucket=BUCKET_NAME, Key=path, Fileobj=bytes_file, Callback=hook(t), Config=TransferConfig(use_threads=False)) works for me.

Is this what your issue was ?

#13

The working console output came from the console of the server running streamlit run, correct. The Streamlit application’s progress bar was stuck.

I added in the disable threading config from the boto3.s3 and it worked! Thanks so much @Wirg