Streamlit is not working with joblib parallel and delayed

Using Streamlit
9

Hi @sarath :wave:

When using @st.cache, you receive the following error:

PicklingError: Could not pickle the task to send it to the workers.

image
image895×739 76.2 KB

Solution

The solution is to decorate double(x) with @st.experimental_memo instead:

import streamlit as st
from joblib import Parallel, delayed, cpu_count
import time
from stqdm import stqdm

@st.experimental_memo
def double(x):
    time.sleep(0.5)
    return 2 * x

def inc(x):
    time.sleep(0.5)
    return double(x)

arguments = range(1, 5)

st.header("Sequential")
last_time = time.time()
results = [inc(x) for x in stqdm(arguments)]
st.write(results)
current_time = time.time()
st.write(current_time - last_time, "seconds")
last_time = current_time

st.header("Parallel")
last_time = time.time()
njobs = cpu_count()-1
results = Parallel(n_jobs=njobs)(delayed(inc)(x) for x in stqdm(arguments))
st.write(results)
current_time = time.time()
st.write(current_time - last_time, "seconds")
last_time = current_time

Output

joblib

Happy Streamlit-ing! :balloon:
Snehan