# Database or CSV files

**URL:** <https://discuss.streamlit.io/t/database-or-csv-files/64005>\
**Category:** Using Streamlit\
**Tags:** cache, discussion\
**Created:** [March 6, 2024, 12:17pm UTC](https://discuss.streamlit.io/t/database-or-csv-files/64005 "2024-03-06T12:17:04Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![optimvs](https://sea2.discourse-cdn.com/streamlit/user_avatar/discuss.streamlit.io/optimvs/32/38344_2.png) [@optimvs](https://discuss.streamlit.io/u/optimvs)\
**Post date:** [March 6, 2024, 12:17pm UTC](https://discuss.streamlit.io/t/database-or-csv-files/64005/1 "2024-03-06T12:17:04Z")

</div>

Hi team, I’ve a big doubt, which is faster to use database or csv files?

I’ve an app to show and filter about 80 csv files and 1Gb data. I need to read every file and filter data to show results. This process takes about 5 minutes and need to refilter in every settings change.

How can I make my app more efficient with so much data?

Thanks a lot

---

<div class="post-metadata">

**Author:** ![dataprofessor](https://sea2.discourse-cdn.com/streamlit/user_avatar/discuss.streamlit.io/dataprofessor/32/44163_2.png) [@dataprofessor](https://discuss.streamlit.io/u/dataprofessor)\
**Post date:** [March 6, 2024, 12:57pm UTC](https://discuss.streamlit.io/t/database-or-csv-files/64005/2 "2024-03-06T12:57:59Z")

</div>

Hi @optimvs

Have you looked into caching your data using `st.cache_data` ([st.cache\_data - Streamlit Docs](https://docs.streamlit.io/library/api-reference/performance/st.cache_data))

Also see this post that suggests using an option for persisting to disk (`st.cache_data(persist="disk")`):

> [@BLOCKING BUG - memory and ephemeral storage leak issue in a Kubernetes deployment](https://discuss.streamlit.io/t/blocking-bug-memory-and-ephemeral-storage-leak-issue-in-a-kubernetes-deployment/61251/4):
>
> @Ian_B To lower the memory usage, there are two quick things you could try: Use [this wheel file](https://core-previews.s3-us-west-2.amazonaws.com/pr-8068/streamlit-1.31.0-py2.py3-none-any.whl) which fixes a memory leak. This leak was fixed last week and will be released with 1.32. Deactivate the backend storage of forward messages in config.toml via (requires at least 1.30):[global] storeCachedForwardMessagesInMemory = false This is a bit of an old artifact that most likely doesn’t have any use. And its a bit of a problem if there is a spike of user sessions. These two aspects might he…

Another area to explore in improving the performance is to determine if the entire data is needed, here are some questions to consider:

- Are all columns needed?
- Or only certain columns are used from the data? If so, you can use the `usecols` parameter of `pd.read_csv`.

Another great read is the blog by @randyzwitch on building performant apps ([6 Tips for Improving Your App Performance | Streamlit](https://blog.streamlit.io/six-tips-for-improving-your-streamlit-app-performance/)), in particular sections 4 and 5 on: \* 4. Remove unused data\* and \* 5. Optimize data storage formats\*

Hope this helps!

---

<div class="post-metadata">

**Author:** ![Goyo](https://sea2.discourse-cdn.com/streamlit/user_avatar/discuss.streamlit.io/goyo/32/28219_2.png) [@Goyo](https://discuss.streamlit.io/u/Goyo)\
**Post date:** [March 6, 2024, 1:19pm UTC](https://discuss.streamlit.io/t/database-or-csv-files/64005/3 "2024-03-06T13:19:56Z")

</div>

Make sure you only read (and parse) the files once, not on every settings change.

---

<div class="post-metadata">

**Author:** ![ferdy](https://sea2.discourse-cdn.com/streamlit/user_avatar/discuss.streamlit.io/ferdy/32/29957_2.png) [@ferdy](https://discuss.streamlit.io/u/ferdy)\
**Post date:** [March 6, 2024, 2:48pm UTC](https://discuss.streamlit.io/t/database-or-csv-files/64005/4 "2024-03-06T14:48:00Z")

</div>

> [@optimvs](#):
>
> which is faster to use database or csv files?

Depends on what you are doing. Eventually you have to test which one is faster in your case.

> [@optimvs](#):
>
> I need to read every file and filter data to show results.

Reading the file is usually the bottleneck. You can solve this by caching. If filtering takes too much time too, try caching the results. Read the caching documents. And test your implementation if there is improvement.

---

<div class="post-metadata">

**Author:** ![system](https://global.discourse-cdn.com/streamlit/original/3X/c/f/cf4c2ee0d938358f82448c5fe4b9c3c91d7ce19c.svg) [@system](https://discuss.streamlit.io/u/system)\
**Post date:** [September 2, 2024, 2:48pm UTC](https://discuss.streamlit.io/t/database-or-csv-files/64005/5 "2024-09-02T14:48:42Z")

</div>

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.
