How to Run a Background Task in Streamlit and Notify the UI When It Finishes

[Victor_Quilon]

Hi everyone,

I’m working on a Streamlit app where I need to run a background task that:

1. Continues running even if the user interacts with the app or navigates to another page.
2. Writes a file to disk upon completion (optional).
3. Sends a notification to the UI when it finishes.

I know that Streamlit runs in a single-threaded environment, but I was wondering if anyone has successfully implemented a background task that doesn’t stop when the user interacts with the UI. My idea was to use threading, asyncio, or even multiprocessing, but I’m not sure which approach would be best.

One possible workaround is to use a subprocess that keeps running independently of Streamlit and writes a file when it’s done. Then, the frontend could periodically check for that file. But ideally, I’d like a more elegant solution that allows the backend to notify Streamlit directly when the task completes.

Has anyone found a good way to do this? Possible Solution Using asyncio and threading

Since Streamlit runs a separate process for each user session, we need to ensure that our background task continues running independently of the session lifecycle. One way to achieve this is by using asyncio with threading to run the task in the background and then notify the frontend when it finishes.

Has anyone tried something similar?
I know there is another discussion about this topic:

Run long running tasks (like embedding of large docs) asynchronously without blocking UI and notify user when finished

Summary I need to run long running tasks (like vector embeddings of large documents) in Streamlit without blocking the user interface. How to properly do this? Additional information I have been reading about using Streamlit asynchronously for the past several days but to me as a newbie (both Streamlit and Python) it is not obvious how to do a simple task: user uploads several large documents that need to be embedded using ChatGPT API calls. This runs a long time and I want to offer the user to…

Looking forward to your thoughts!

Thanks!

[EmilienF]

@Victor_Quilon

One possible workaround is to use a subprocess that keeps running independently of Streamlit and writes a file when it’s done. Then, the frontend could periodically check for that file. But ideally, I’d like a more elegant solution that allows the backend to notify Streamlit directly when the task co

Hi Victor

I’ve used the file + subprocess solution you’re mentionning, on my end on carburoam, an open source app which exposes daily price of gas stations in France

Have a look at function trigger_etl from homepage triggered by visitors here and function main_etl here

I had to implement a mecanism to refresh the local database of prices.
This was pretty straightforward to do wit this solution, and even if it doesn’t seems best practice to check file and do it this way, honestly it does works very well for a hobby/fun. project

If you’re looking to deep dive on the subject, I’ve written a blog post explaining details about this implementation and the workflow (ensure another process hasn’t started, monitoring, and so on..)

NB:Make sure to use python executable during subprocess calls, you will avoid some debugging issue or python env related issue when comparing local env during developement and Streamlit cloud one

If I had to pick some production grade orchestrator I would have used python rq along a redis pubsub service.
I’ve already battle tested it on production project but here with streamlit Cloud you couldn’t connect any external services easily. it’s only an option if you are using Render for instance

[REPNOT]

Streamlits built-in fragments and caching features might be a possible solution, but Streamlit does provide guidance on multithreading in the documentation which is listed below. I’ve found Fragments to be highly useful and would recommend atleast reviewing the execution flow page just for future reference.

Fragments, Caching, and Session State in Streamlit

Use core features to work with Streamlit's execution model - Streamlit Docs

Caching overview - Streamlit Docs

Add statefulness to apps - Streamlit Docs

Multithreading in Streamlit

Threading in Streamlit - Streamlit Docs