How do we display the prints inside children processes in a joblib parallel job onto the streamlit output?

🎈 Using Streamlit
1

We can see the prints inside the children processes on a python terminal, but when we use st.write() or overwrite.write(), nothing is displayed on the streamlit output.

Specifically, we build a timer inside the sklearn.joblib parallel helper function. It will print how much time it has taken, and estimate how much time it may still take. This timer message is printed successfully on the python terminal using print(), but not displayed if we use streamlit (st.write() or overwrite.write()).

Any ideas?

2

Hi @gopeaks

Thanks for the question, however it is difficult to see where the possible error may be without a reproducible code snippet that others in the community can see and help. Thus, could you share a reproducible working example of your code.

Best regards,
Chanin

3

Thank you, sure. below is our codes.

== python code for python terminal:

from joblib import Parallel, delayed, parallel_backend
import time

def square(i, iterations):
start = i + iterations
res =
start_time = time.time()
for j in range(start, start+iterations):
res.append(jj)
time.sleep(1)
if i == 0:
time_elapsed = time.time()-start_time
time_left = time_elapsed(iterations-(j+1-start))/(j+1-start)
print(f"{j+1-start}/{iterations} done in {time_elapsed}s. {time_left}s to go.“, end=”\r")

if i == 0:
    print(f"\nFinished in {time.time()-start_time}s")
return res

def main():
n_jobs=2
iterations = 3

print("Start computing...")

results = Parallel(n_jobs=n_jobs, verbose=0, backend='multiprocessing')(\
    delayed(square)(i, iterations) for i in range(n_jobs))


print(f"results: {results}")

if name == “main”:
main()

== Streamlit py script for doing the same thing:

from joblib import Parallel, delayed, parallel_backend
import time

import streamlit as st
st.set_page_config(layout=‘wide’)
from io import BytesIO

progress_bar = st.progress(0)
overwrite = st.empty()

def square(i, iterations):
start = i + iterations
res =
start_time = time.time()
for j in range(start, start+iterations):
res.append(jj)
time.sleep(1)
if i == 0:
progress_bar.progress(round(100(j+1-start)/(iterations)))
time_elapsed = time.time()-start_time
time_left = time_elapsed*(iterations-(j+1-start))/(j+1-start)
overwrite.write(f"{j+1-start}/{iterations} done in {time_elapsed}s. {time_left}s to go.")

if i == 0:
    st.write(f"\nFinished in {time.time()-start_time}s")
return res

def main():
n_jobs=2
iterations = 3

st.write("Start computing...")

results = Parallel(n_jobs=n_jobs, verbose=0, backend='multiprocessing')(\
    delayed(square)(i, iterations) for i in range(n_jobs))


st.write(f"results: {results}")

if name == “main”:
main()