How to separate fetching data from user interaction?

Summary

I am developing a web app using streamlit which is doing pretty straightforward stuff: 1) fetching data from SQL; 2) user enters some inputs; 3) some calculations based on the data fetched and user inputs, and output results.

The problem is fetching data and some of the calculations take long time. I used @st.cache_data to cache functions of fetching data and calculation. But after ttl (say 1 hour), at least 1 user would trigger a new round of fetching data and calculation, and it will still take long time, which gives this user bad experience.

What I want to achieve is no user need to wait for the app to fetch data and do calculation (the first time when the app is launched does not count). I am thinking having a separate process or job or whatever that can fetch data and do calculations independently every 1 hour, and it can cache data and results. And the web app will just source the cashed data and results from that process, so no user would wait for it fetching data or doing calculation. Is there a way to do that? Or is there any better way to never wait the app to fetch data and do calculation? Thank you.

To improve the user experience and avoid long wait times for data fetching and calculations in your Streamlit web app, you can implement a background process or job to periodically fetch data and perform calculations independently. Here’s a suggested approach:

  1. Separate the data fetching and calculation logic into a separate Python script or module. This script will be responsible for fetching data and performing the necessary calculations.

  2. Create a background process or job that runs independently of the Streamlit web app. This process will be scheduled to execute the data fetching and calculation logic at regular intervals, such as every hour.

    You can achieve this by using a task scheduling library like schedule or APScheduler. These libraries allow you to define tasks that run periodically. In your case, the task will trigger the data fetching and calculation logic.

  3. When the background process or job completes the data fetching and calculations, store the results in a persistent data store or cache. This could be a database, file system, or an in-memory cache like Redis.

  4. In your Streamlit web app, instead of directly fetching data and performing calculations, you will query the results from the persistent data store or cache. This way, the web app can access the precomputed results without waiting for the data fetch and calculations to complete.

    Make sure to handle cases where the web app requests data that has not yet been fetched or calculated by displaying appropriate loading indicators or error messages.

By decoupling the data fetching and calculation process from the web app and using a background process or job to handle these tasks independently, you can ensure that users don’t need to wait for the data fetch and calculations to complete. They can access the precomputed results immediately from the persistent data store or cache, providing a better user experience.

Note: You need to carefully design the synchronization between the background process and the web app to handle cases where the web app requires the most up-to-date data. This can be achieved by incorporating appropriate mechanisms, such as cache invalidation or updating the cache when new data is fetched.

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.