How to avoid reloading of pyplot figures?

I am working on an app that takes sequence of user inputs and renders pyplot figures. I am using session_state to avoid redundant computations, however I am not able to do the same with the matplotlib figure that I am creating. I tried storing the rendered figure in sessions and then returning it if nothing is related to it is changed, however even then the st.pyplot(fig) command takes about 10-15 seconds to render the same image. How do I avoid this?