Database or CSV files

[dataprofessor]

Hi @optimvs

Have you looked into caching your data using st.cache_data (st.cache_data - Streamlit Docs)

Also see this post that suggests using an option for persisting to disk (st.cache_data(persist="disk")):

BLOCKING BUG - memory and ephemeral storage leak issue in a Kubernetes deployment

@Ian_B To lower the memory usage, there are two quick things you could try: Use this wheel file which fixes a memory leak. This leak was fixed last week and will be released with 1.32. Deactivate the backend storage of forward messages in config.toml via (requires at least 1.30):[global] storeCachedForwardMessagesInMemory = false This is a bit of an old artifact that most likely doesn’t have any use. And its a bit of a problem if there is a spike of user sessions. These two aspects might he…

Another area to explore in improving the performance is to determine if the entire data is needed, here are some questions to consider:

• Are all columns needed?
• Or only certain columns are used from the data? If so, you can use the usecols parameter of pd.read_csv.

Another great read is the blog by @randyzwitch on building performant apps (6 Tips for Improving Your App Performance | Streamlit), in particular sections 4 and 5 on: * 4. Remove unused data* and * 5. Optimize data storage formats*

Hope this helps!