Best (current) practice for spawing a process to run independent of Streamlit?

[ZKLO]

Solved, subprocess works perfectly but had to use:

if st.button("Process data"):
  
  subprocess.Popen(['python','processor.py',selected_file])

I know similar issues has been posted before, but I couldn’t relate it to what I am trying to achieve.

Let’s say a user has a st.button they can click. Upon clicking they tell the underlying server to start processing a large file that might take ~24 hours to process and they might get a brief st.success message “Processing has started, please come back later”.

*After clicking the button the user might still want to use the app for other tasks and will come back the next day to check the results of the first file. *

What is the current best practice to spawn in underlying process and keep using streamlit (with having the constant running)? This approach underneath just seems to leave the current streamlit window running:

sudo code:

```

• import subprocess*
• import os*

raw_files = os.listdir(“raw_data/”)
selected_file = st.selectbox(“Select file to process”,raw_files )
if st.button(“Process data”):

• subprocess.run(f"python processor.py {selected_file}", shell=True)*
  ```

Any ideas?

[asehmi]

Hi,

You can use this approach here, which spawns a Flask server, and calls its /foo endpoint from Streamlit. You could replace that endpoint implementation with your long running process (LRP). Via an additional /status endpoint you can occasionally request the status of the LRP from Streamlit. Your LRP could occasionally drop a status message file as it does its work for /status to read or maybe your LRP can be queried directly.

For something more robust run and monitor your LRP, use a job queue system, and call their APIs from Streamlit to get job statuses. Examples (for Python) that come to mind are:

• GitHub - rq/rq: Simple job queues for Python
• Using Faktory with Python | Mike Perham
• There are many more…

HTH,
Arvindra