Can we use PySpark dataframes instead of Pandas dataframes in backend while developing app using Streamlit?
Hello @Santhosh_Kumar
Anything you can run in Python should run with Streamlit, since Streamlit only really displays a web frontend for your Python code. As long as you convert Pyspark back to small Pandas Dataframes before displaying with Streamlit, youβll be good. For example:
- Pyspark Streamlit demo for university (github.com)
- How I keep my master students engaged during Python lectures - YouTube
Have a nice day ,
Fanilo
Hello, if my pyspark DF have more than 10k rows, I need to use .toPandas() to convert it before if I want to show all rows. Why? Any other solution to not to use .toPandas()?