Embed a 3D (VTK) visualization app and drive it with Streamlit

[nantille]

Hello,

I’d like to know if there is a way to use Streamlit like PyQt, as a way to embed another 3D viewer that does things that Streamlit or JS cannot handle like a C++ renderer would. Think volume rendering, slicing, etc.
For instance, I’m using ipyvtklink which derives from ipycanvas to stream images from the vtk app straight to the html canvas of a web page.

The viewer I’ve developed for computational neuroscience is based on vtk and vedo. The viewer works within Jupyter notebooks thanks to ipyvtklink.

Now we would like to add a web UI around it with the possibility to easily plot stats.

Btw, I’m not interested in vtk-js as it’s not on par with vtk at the moment, hence the streaming question, which is also interesting for other cpp 3D renderers.

[banesullivan]

Hi there, I’m one of the developers of PyVista and I’m the creator of ipyvtklink! I’ve been toying with streamlit recently and I had this same idea and started prototyping this to see whether it is possible with the new Components API.

Unfortunately, I don’t think it is possible/feasible to implement an ipyvtklink-style component in Streamlit with the current state of the Components API

ipyvtklink works by, like you describe, streaming an image/screenshot of the vtkRenderWindow to a canvas in the HTML of the client. Then, the mouse/interactor events of that canvas are captured and sent back to Python. It’s that second piece of capturing the interaction events that is so critical to get this working; if you can efficiently send the mouse events back to Python (to the vtkRenderWindow), then you can push those events into vtk’s event stream and produce a new thumbnail to stream back to the client achieving decent interactivity.

In streamlit’s new Component API, there appears to only be one mechanism for sending data back to Python from the web client: through the Streamlit.setComponentValue TypeScript function. Unfortunately, when Streamlit.setComponentValue sends data back to Python, it re-executes the Python script from top to bottom. This re-execution part is not needed for this use case and causes the interactivity to pretty much fail.

What we need is a mechanism to send data back to Python without re-execution so that we can push the data to the vtkRenderWindow event stream (or better yet, in a background thread) and keep the component its existing state, rather than effectively recreating it on each Streamlit.setComponentValue call.

I’ve done quite a bit of initial prototyping on this widget, but the re-execution aspect of Streamlit.setComponentValue blocks me from creating anything halfway useable.

For now, I’m going to pursue creating a one-direction component that simply serializes a vtkRenderWindow (PyVista Plotter) into a VTK.js or three.js scene. This would be inline with the post in Is it possible plot 3D Mesh file or to add the VTK/ Pyvista window inline streamlit?

Disclaimer: I literally started learning streamlit today, so my understanding of how the Component API works and what should be/is possible could be totally wrong. I look forward to any help!!

[andfanilo]

@banesullivan

Yep you got everything right on the current status of components.

Even for me on drawable canvas, returning the drawing anytime the user draws something takes some time with all the serializing/deserializing going on. So I added a button to explicitly send data back to Python (click = call setComponentValue) instead of returning data on every interaction. The bi-directional on-demand could be a good compromise for you too.

Also maybe for inspiration you can read the design of streamlit-webrtc, I don’t specifically remember what’s in there that could help, I just vaguely remember about an independent background asyncio thread, maybe that can help too on the pushing to an independent event stream without relying on the streamlit execution model (did not thouroughy think about it, this is just a fleeting idea)…

Happy to see you’re trying to work this through
Fanilo

[banesullivan]

Thanks, @andfanilo! I’m definitely going to look into streamlit-webrtc a bit as that seems like my best lead at the moment.

[banesullivan]

I recently opened a PR in streamlit to enable port proxying:

Add opt-in port proxying via jupyter-server-proxy

If this is approved and merged, I can likely use something like imjoy-rpc or wslink within the existing Components API in streamlit to accomplish this

This would make it not so difficult to have server-side 3D rendering where mouse events in the browser are sent to the server which produces a screenshot and streams that back to the client

[andfanilo]

The PR and solution is nice, and would definitely help a lot of other use cases

From my experience as a long-time lurker of Streamlit developments, this type of PR may take some time to get reviewed by both Streamlit and Streamlit Cloud product teams (as they will probably think about how much external processes they want to enable in Cloud) + the dev team. I can push it a little internally but it’ll still require a small bit of patience (btw lemme ping @jrieke on this PR about proxying specific queries to a locally hosted web server, it solves different issues around querying a locally hosted localtileserver, it could help components development too, or using a local FastAPI for specific OAuth work)

Let’s see how far we can get that too.
Fanilo

[napoles3d]

Hi @banesullivan, Great work with pyvista and ipyvtklink!
I tried pyvista in streamlit and it works at least locally, check this example (one-directional):

import streamlit as st
import streamlit.components.v1 as components
import pyvista
from pyvista import examples
mesh = examples.load_uniform()
pl = pyvista.Plotter(shape=(1,2))
_ = pl.add_mesh(mesh, scalars='Spatial Point Data', show_edges=True)
pl.subplot(0,1)
_ = pl.add_mesh(mesh, scalars='Spatial Cell Data', show_edges=True)
pl.export_html('pyvista.html')  # doctest:+SKIP

option=st.sidebar.radio('Pyvista',('On','Off'))
if option=='On':
  HtmlFile = open("pyvista.html", 'r', encoding='utf-8')
  source_code = HtmlFile.read() 
  components.html(source_code, height = 500,width=500)

you can find the code in this repo

This looks like this:

Also works on streamlit cloud:
https://share.streamlit.io/napoles-uach/pyvista_streamlit/main/app.py

Hope it helps!
Regards

[BenBen]

Thanks, for your investigation :-).
Is there any chance the bidirectional can be solved ?

chhers

[rbedford]

Hi there,

Sorry for reopening such an old thread, I’m wondering if there are any updates on this? We also have a streamlit application which opens visualisation tools in vedo. Locally, we can just open a new vedo window with the vtk backend. However, we want to deploy our app on our server (currently using docker for deployment).

We’ve been thinking of a number of solutions but none of them seem quite right - such as opening a VNC server, or using SSH X forwarding. We want the interactive 3D plot to be available for the user, either embedded in the app or in a new window (we really don’t mind if we can find a working solution), but opened within the streamlit app without them having to, for example, login to the server with SSH.

Vedo supports a few different backends: vtk, k3d, ipyvtklink, and trame. If anyone can think of a way to display one of these in streamlit please let me know. Alternatively, we could try and rewrite our application using plotly 3D rendering, but this would require a lot more work so we’re keen to see if there’s another solution!

Thanks!

[edsaac]

I built stpyvista (GitHub - edsaac/stpyvista: 🧊 Show pyvista 3D visualizations in streamlit) to embed 3D pyvista visualizations into streamlit apps, which includes support for VTK files. It is not a bi-directional component but for visualization it has been good enough.

Here is the demo app:

stpyvista

Hope it helps.

[rbedford]

Thanks this looks great I will take a look and let you know how we get on!