Reanding parquet file from S3 using Dask Dataframe

caiosoter · December 27, 2023, 7:23am

Hey, guys since my dataset is very large to fit in memory, I have been trying to use Dask to manipulate the data to get a new dataframe that could fit in memory. However, I am getting a erro that I could not resolve…
I am using this function to read the data:

And I am getting the erro below we I use the data from above, inside the functions below

My reppo

github.com

caiosoter/Book_Adviser/blob/main/main.py

import streamlit as st
import dask.dataframe as dd
import pandas as pd
import re
import joblib
import os
from io import BytesIO
import tempfile
import scipy.sparse as ss
import numpy as np
import boto3
import logging
from sklearn.metrics.pairwise import cosine_similarity
st.set_page_config(page_title="Book Advisor", page_icon=":book")

@st.cache_resource
def get_s3_client():
   s3 = boto3.client('s3', 
         aws_access_key_id=st.secrets["AWS_ACCESS_KEY_ID"],
         aws_secret_access_key=st.secrets["AWS_SECRET_ACCESS_KEY"])

This file has been truncated. show original

system · June 24, 2024, 7:24am

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Read large files from s3 using st.connection Using Streamlit discussion	3	208	June 16, 2024
How do i cache multiple file reads from s3? Using Streamlit cache	2	1487	November 19, 2021
Streamlit version 1.23.1: WebSocket Timeout Issue Using Streamlit	9	4170	February 13, 2024
Upload a source file from my local and deploy the streamlit app on the browser Using Streamlit file-upload , streamlit-cloud	7	2254	April 7, 2024
St.cache_data warning on AWS Deployment cache , pandas	6	428	September 7, 2023

Reanding parquet file from S3 using Dask Dataframe

Related topics

Hello there 👋🏻

Cookie settings

Strictly necessary cookies

Performance cookies

Functional cookies

Targeting cookies