multiprocessing.dummy.Pool

Using Streamlit
1

If I load a pickle file from inside a multiprocessing.dummy.Pool, I need to rerun the streamlit twice every time any code changes. The exception I get is:

AttributeError: Can't get attribute 'LogDirectory' on <module 'sandbox.config' from '/root/src/sandbox/sandbox/config.py'>
Traceback:
  File "/usr/local/lib/python3.6/dist-packages/streamlit/ScriptRunner.py", line 324, in _run_script
    exec(code, module.__dict__)
  File "/.../runner.py", line 98, in <module>
    exec(filebody, {})
  File "<string>", line 317, in <module>
  File "<string>", line 55, in main
  ...
  File "/XXX.py", line 124, in all
    return filter(None, self.pool.map(load, filenames))
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/XXX.py", line 79, in load
    return pickle.load(open(filename, "rb"))

Is there a way to fix this? I am not using any st.cache. The attribute it can not get is of a class that isn’t changing and has nothing to do with the changed file. It doesn’t matter what file I change, that same error comes. LogDirectory is just a simple class, nothing fancy. sandbox.config it is in a file called sandbox/config.py.

2

Hi @Zach_Dwiel,
thanks for posting. Can you share a simple app that reproes this issue? By doing some simple testing with multiprocessing.dummy and pickle I could not find the problem you are describing.

Matteo

3

It looks like this problem may be related to this bug: https://github.com/streamlit/streamlit/issues/1511 . The LogDirectory was being used something like this:

from sandbox import config
...
config.LogDirectory(...)

It looks like the hot reloading in cases like these is not working and may have caused related problems with multiple objects being defined by the same name?