How to upload large files using st.file_uploader

serdar_bay · January 5, 2023, 1:29pm

I am trying to upload a binary file of size 10gb using st.file_uploader , however, I get the following error message. In fact, I get the same error message pretty much when I am trying to upload any file above 2gb, i.e I didn’t get this error message when I uploaded 1.2gb file.

I have set my file capacity to 10gb in config file.

As a matter of fact, the file loading shows as it should, then the file appears to be uploaded for a very short time (maybe a second), however, then the attachment disappears along with file_uploader widget itself, and the following message pops up.

snehankekre · January 11, 2023, 11:23am

Hi @serdar_bay

Thank you for creating this post! I think you’ve run into a bug with st.file_uploader. I was able to reproduce the reported behavior and have submitted a bug report in the Streamlit repo. Please consider upvoting the issue on GitHub for it to be prioritized by our engineers

github.com/streamlit/streamlit

"ValueError: Value out of range" when uploading large files (~>= 4.5GB) with `st.file_uploader`

opened 11:20AM - 11 Jan 23 UTC

snehankekre

type:bug feature:st.file_uploader status:needs-triage

### Checklist - [X] I have searched the [existing issues](https://github.com/st…reamlit/streamlit/issues) for similar issues. - [X] I added a very descriptive title to this issue. - [X] I have provided sufficient information below to help reproduce this issue. ### Summary Uploading a large file with `st.file_uploader` can reliably lead to a: ``` ValueError: Value out of range <uploaded-byes> File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 565, in _run_script exec(code, module.__dict__) File "/Users/skekre/Downloads/streamlit-misc/file-uploader-size.py", line 3, in <module> f = st.file_uploader("Upload a file") File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/streamlit/runtime/metrics_util.py", line 311, in wrapped_func result = non_optional_func(*args, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/streamlit/elements/file_uploader.py", line 338, in file_uploader return self._file_uploader( File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/streamlit/elements/file_uploader.py", line 418, in _file_uploader file_uploader_state = serde.serialize(widget_state.value) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/streamlit/elements/file_uploader.py", line 108, in serialize file_info.size = f.size ``` where "large" in my case was approximately close to or greater than 4.5GB. I'm not sure whether the specific size after which this error is thrown is machine dependant. The user who first [reported](https://discuss.streamlit.io/t/how-to-upload-large-files-using-st-file-uploader/35654) this on the Streamlit forum encountered the error for files >= 2GB. ### Reproducible Code Example ```Python import streamlit as st f = st.file_uploader("Upload a file") if f is not None: st.write(f.name) st.write(f.size) ``` ### Steps To Reproduce 1. Create a `.streamlit/config.toml` file to set the upload limit to 4.5 GB: ```toml [server] maxUploadSize = 4500 ``` 2. Create a 4.4 GB file: ```sh truncate -s 4280m example.file ``` 3. Run the above app and upload `example.file` to the file uploader widget ### Expected Behavior As the uploaded 4.4 GB file is under the 4.5 GB limit set in `.streamlit/config.toml`, one would expect: 1. to see the upload successfully complete 2. the file uploader widget to be visible after the upload 3. see the output of the filename and filesize displayed ### Current Behavior 1. Once the progress bar on the file uploader widget hits the maximum, the app greys out for a few seconds while the Running man icon is displayed in the top right corner 2. Immediately after that, the file uploader widget disappears, leaving the user with the following error: ``` ValueError: Value out of range: 4487905280 Traceback: File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 565, in _run_script exec(code, module.__dict__) File "/Users/skekre/Downloads/streamlit-misc/file-uploader-size.py", line 3, in <module> f = st.file_uploader("Upload a file") File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/streamlit/runtime/metrics_util.py", line 311, in wrapped_func result = non_optional_func(*args, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/streamlit/elements/file_uploader.py", line 338, in file_uploader return self._file_uploader( File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/streamlit/elements/file_uploader.py", line 418, in _file_uploader file_uploader_state = serde.serialize(widget_state.value) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/streamlit/elements/file_uploader.py", line 108, in serialize file_info.size = f.size ``` ![image](https://user-images.githubusercontent.com/20672874/211792715-25c255b5-ff1a-4bdc-a24b-6739b85cd42e.png) ### Is this a regression? - [ ] Yes, this used to work in a previous version. ### Debug info - Streamlit version: 1.16.0 - Python version: 3.9 - Operating System: macOS - Browser: Chrome - Virtual environment: NA ### Additional Information First reported in https://discuss.streamlit.io/t/how-to-upload-large-files-using-st-file-uploader/35654 ### Are you willing to submit a PR? - [ ] Yes, I am willing to submit a PR!

Best,
Snehan

serdar_bay · January 11, 2023, 11:43am

Thanks @snehankekre for the update and reporting the bug in the repo!