Read large files from s3 using st.connection

Using Streamlit
1

Hi folks

I am facing an issue with my connection between Streamlit and aws s3
here I share my code snipped:

conn = st.connection('s3', type=FilesConnection)
df = conn.read("s3://bucket/file.parquet", input_format='parquet')

I mostly use parquet files because of the possibility of filtering and selecting columns before even loading the full data to a variable. Is there any way I could do these queries using conn.read too?

I will appreciate your help!
thanks!

2

I don’t think so. You can use the open method and pass the output wherever a file-like object is expected.

3

you just saved my life mate

4

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.