I’ve been playing around with this some more and I think I’ve isolated the issue.
Instead of using a process pool, I tried a thread pool and the problem doesn’t seem to present. Although, my app runs noticeably slower…
I believe this comes down to how Streamlit is killing an in-process script run. Killing the in-process script run appears to be quite abrupt with no signals to the running process (at least, none that Python would just natively interrupt for). Normally, I imagine killing the script this way is probably fine but if a multiprocessing.pool.Pool has been instantiated, this just leaves the child processes hanging. I think this then leads to a corrupted session since the session is seemingly blocked from resetting or something because of the hanging child processes.
So, I think this means Streamlit does not really support spawning children processes within a script if that script can be rerun within the same session. I’ll try to drop this in a github issue later.