Hi everyone!
I created a streamlit app in which, at first, some data is downloaded with get_data() function, then some manupulations with the dataset are done and finally some plots and tables with results are displyed to the user. For get_data function I use st.cache to prevent data loading every time the user chahges widgets in the app.
I’m running streamlit in docker container, so to make changes in the app I need to stop the app and cntainer, do changes and then run docker again. When I had very few users I could easilly stop the app, make some changes in the code and then rerun the app. After rerun get_data usually consumed 4-5 GB of RAM memory. And while the app was running memory consumption never exceeds 5-6 GB.
After a while number of users of the app increased and now I experience problems with rerunning the app. I decided to improve my app, stopped the container, did some changes and rerun the container. Accidentally I experience huge memory usage, more than 30 gb and my linux server has only 32 gb of RAM.
After a while, I guess when all data for all users was loaded, memory again returned to 5-6GB. I monitor memory usage wia htop command.
I guess that the problem is that the app is opened at the same time by many users and when I rerun the app, get_data function starts to load data in all opened URLs… As a result instaed of using 5 GB it uses 30+ GB.
Please, can anyone tell if my guesses are correct? And what could be done to decrease memoty usage while loading? I guess that many users have the app opened in the background mode, I mean they just don’t close the URL after they finished working with the app.
I tried to romove st.cache but this did not helped.
Maybe there is a way to kill opened URLs? Or put data loading in all these opened URLs in a queue?
I’m using streamlit==1.3.1