Unable to refresh data from S3

Using Streamlit
1

Hi - I am unable to refresh data from S3.

from pathlib import Path
from s3fs import S3FileSystem as s3

# @st.cache(ttl=2)
def get_run_names():
    """List existing run names"""
    BASE_PATH = "[INSERT_S3_DIRECTORY_HERE]"
    run_names = [Path(file).stem for file in s3().ls(BASE_PATH)]
    if st.button("Click to Refresh!"):
        del run_names
        run_names = [Path(file).stem for file in s3().ls(BASE_PATH)]
    return run_names

When I click the “Click to Refresh!” button, I still get the same list of files even though the directory has changed and when I run this gist in the terminal, it gives me the proper directory list.

Please help :slight_smile:

2

Bump

3

Hello @Laksh_Aithani,

Just to confirm that detail, you commented the @st.cache line during your tests?
Does the list update when you press R to refresh the page? And when you press F5?

Also, when you click the update button, the whole script re-runs. That means the first run_names line is executed, and as you pressed the button, the second run_names line is executed as well.

4

Hi @okld I have tried with and without st.cache, nothing works.

Yes, when I press refresh, or when I reset the cache, it still doesn’t work.

The only thing that works is shutting the app down in the terminal and restarting streamlit run app.py again!

This is what is confusing. It still does not update!

5

Well, the only explanation I have is that S3FileSystem caches your result to avoid sending too many requests. I’ve come across the clear_instance_cache() in S3Fs documentation. Maybe you could try calling that after your ls(). If it doesn’t work, I saw other options related to caching in the same page. You could try to play with those as well.

1 Like
6

s3.S3FileSystem().ls(path, refresh=True) solved it!

Thanks a lot @okld :slight_smile:

3 Likes