Can we use PySpark dataframes instead of Pandas dataframes in backend while developing app using Streamlit?
Hello @Santhosh_Kumar
Anything you can run in Python should run with Streamlit, since Streamlit only really displays a web frontend for your Python code. As long as you convert Pyspark back to small Pandas Dataframes before displaying with Streamlit, you’ll be good. For example:
- Pyspark Streamlit demo for university (github.com)
- How I keep my master students engaged during Python lectures - YouTube
Have a nice day 
,
Fanilo
3 Likes
As of 1.16.0 Streamlit ships with native support for PySpark DataFrames. Yay!!!
1 Like
Hello, if my pyspark DF have more than 10k rows, I need to use .toPandas() to convert it before if I want to show all rows. Why? Any other solution to not to use .toPandas()?
This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.