Not sure if it’s there! But I was wondering if anyone has managed to build PySpark based application using Streamlit ?
Hi @Shreyas_Raghavan, welcome to the Streamlit community!
For pyspark, the eventual result can be transferred to the local machine as a pandas dataframe, which Streamlit supports natively. So the question is less about whether Streamlit supports pyspark, but rather whether pyspark is the appropriate backend for your app and the user experience.
One thing to keep in mind is that Streamlit will re-run the script top-to-bottom on user input, so it’s possible that you could create dozens of Spark jobs as the user is changing the inputs. We don’t have any mechanism specifically to tell pyspark how to evaluate the code, just that it needs to be evaluated.
Best,
Randy