@sridarr are you actually trying to load 50GB into a browser?
I assume that’s not the case. So it must be that you just want to transfer or read 50GB raw data from DataBricks to whatever server you’re running streamlit on then let your DS’s or whoever run some Python scripts against it and display some of the results in Streamlit?
The details of how you’re processing the data may matter here. If you just need an example of how to get 50GB (or whatever size) data from DB to a server via Python I can find that for you but I assume you probably already have it. So maybe it’s just that you were thinking of this as a Streamlit operation when really the data pull doesn’t need to be a Streamlit step.
Are they trying to read this data dynamically with it updating all the time and always wanting the Streamlit app hitting the most recent version of the data? Maybe you can give an example. If the real use case is too secretive you can just explain by analogy like “the 50GB is a large dataset of transactions, the Streamlit app will use menus to let users graphically create DB queries which will then be summarized in a word cloud and a tabular table. The data is refreshed daily. The upstream database type is Snowflake.”.