Help with loading the pickle file in my app

[Saniya]

I have been trying to deploy an app on the cloud but it just doesn’t happen… Two days ago it was only not able to read my pickle file and threw an error but today the entire app crashed. Could anyone help me with this?

Also I’m a beginner at this so I don’t have knowlegde of things in depth.

[edsaac]

That pickle file is too large for git alone, you need git-lfs. I think adding git-lfs to packages.txt is enough to get in installed in Streamlit Cloud. However, everything hinges on whether you are managing that large file with GitHub:

About Git Large File Storage - GitHub Docs

[Saniya]

I have tried using lfs too… When I upload my file, it gets condensed to 3 lines of code while the size remains same, this is how lfs shows up when I tried to use git push --force

[edsaac]

You are exceeding the per-file limit of 2GB for Github Free.

---

If you don’t need to actually keep track of changes to that file, you could serve it from somewhere else and download it once when the app runs for the first time. Something like:

import os
import shutil

import requests
import streamlit as st

bigfile_name = "my_file.pkl"

if not os.path.exists(bigfile_name): 
    with st.spinner("Downloading..."):
        # URL to a publicly accessible link
        url = "https://drive.google.com/uc?id=<YOUR-FILE-ID>&export=download&confirm=yes"

        # Download and save
        with requests.get(url, stream=True) as r, open(bigfile_name, "wb") as f:
            shutil.copyfileobj(r.raw, f)


st.title("My App")
st.write(f"Size of `{bigfile_name}` is {os.path.getsize(bigfile_name) / 1024:.1f} KB")  # Show file size

[Saniya]

I see, thanks for the response, I’ll try it and update if it works!

[Saniya]

@edsaac

bigfile_name = "my_file.pkl"

if not os.path.exists(bigfile_name): 
    with st.spinner("Downloading..."):
        # URL to a publicly accessible link
        url = "https://drive.google.com/uc?id=<YOUR-FILE-ID>&export=download&confirm=yes"

        # Download and save
        with requests.get(url, stream=True) as r, open(bigfile_name, "wb") as f:
            shutil.copyfileobj(r.raw, f)

Thanks a lot for your response, I have tried it and it removed the error, however, another one occured relating to the file name:

Could you help me with this?

[edsaac]

It’s hard to tell from a screenshot only (Using Streamlit: how to post a question in the Streamlit forum)… sim is a string and it has less than idx characters

---

This link is broken:

@Saniya

Repo: https://github.com/saniyansari/anime-recommender-system

[Saniya]

I’m trying to paste the working link but it’s not allowing me to, please remove the extra h to view it:

link: anime-recommendation-system/app.py at main · saniyansari/anime-recommendation-system · GitHub

[edsaac]

sim is just a string with the name of the pickle file. You might want to uncomment the line where you unpickle and assign the contents of that file to the sim variable.

app.py

[Saniya]

tried that and there’s errors again