Is there any support for PySpark from Streamlit?

Can we use PySpark dataframes instead of Pandas dataframes in backend while developing app using Streamlit?

Hello @Santhosh_Kumar

Anything you can run in Python should run with Streamlit, since Streamlit only really displays a web frontend for your Python code. As long as you convert Pyspark back to small Pandas Dataframes before displaying with Streamlit, you’ll be good. For example:

Have a nice day :balloon: ,
Fanilo

3 Likes

As of 1.16.0 Streamlit ships with native support for PySpark DataFrames. Yay!!! :balloon:

2 Likes

Hello, if my pyspark DF have more than 10k rows, I need to use .toPandas() to convert it before if I want to show all rows. Why? Any other solution to not to use .toPandas()?