Embed a 3D (VTK) visualization app and drive it with Streamlit

Hi there, I’m one of the developers of PyVista and I’m the creator of ipyvtklink! I’ve been toying with streamlit recently and I had this same idea and started prototyping this to see whether it is possible with the new Components API.

Unfortunately, I don’t think it is possible/feasible to implement an ipyvtklink-style component in Streamlit with the current state of the Components API :disappointed:

ipyvtklink works by, like you describe, streaming an image/screenshot of the vtkRenderWindow to a canvas in the HTML of the client. Then, the mouse/interactor events of that canvas are captured and sent back to Python. It’s that second piece of capturing the interaction events that is so critical to get this working; if you can efficiently send the mouse events back to Python (to the vtkRenderWindow), then you can push those events into vtk’s event stream and produce a new thumbnail to stream back to the client achieving decent interactivity.

In streamlit’s new Component API, there appears to only be one mechanism for sending data back to Python from the web client: through the Streamlit.setComponentValue TypeScript function. Unfortunately, when Streamlit.setComponentValue sends data back to Python, it re-executes the Python script from top to bottom. This re-execution part is not needed for this use case and causes the interactivity to pretty much fail.

What we need is a mechanism to send data back to Python without re-execution so that we can push the data to the vtkRenderWindow event stream (or better yet, in a background thread) and keep the component its existing state, rather than effectively recreating it on each Streamlit.setComponentValue call.

I’ve done quite a bit of initial prototyping on this widget, but the re-execution aspect of Streamlit.setComponentValue blocks me from creating anything halfway useable.

For now, I’m going to pursue creating a one-direction component that simply serializes a vtkRenderWindow (PyVista Plotter) into a VTK.js or three.js scene. This would be inline with the post in Is it possible plot 3D Mesh file or to add the VTK/ Pyvista window inline streamlit?

Disclaimer: I literally started learning streamlit today, so my understanding of how the Component API works and what should be/is possible could be totally wrong. I look forward to any help!!

1 Like