3D data visualization: point clouds and triangular meshes

luca · December 3, 2019, 10:27pm

Hello everyone,

I’m really excited about this project!

I work in the geometry processing area, I’d love to use streamlit to visualize 3D data (point clouds and triangular meshes) and functions defined over such 3D data.

Some examples of the visualizations I’m speaking of are the following:

Usually I use MATLAB to do this. When I have to use python (e.g. when using PyTorch) I use open3D or this library.

Can I do this kind of visualizations using streamlit? I’m sure a lot of people would love this.

I understood that streamlit has a wrapper over deck_gl, which supports a PointCloudLayer.
I’d like to start by using this layer to plot point clouds.

I found an example of usage of this layer in javascript here.

However, I’m not familiar with javascript or deck_gl and I don’t seem to be able to make it work.

This is a sketch of the code. The strange data format is what seems to be required by the deck_gl API, but I might be wrong.

vertices = mesh.v
normals = mesh.estimate_vertex_normals()

data = [{
    'position': vertices[i],
    'normal': normals[i],
    'color': np.abs(vertices[i]) / 256,
} for i in list(range(mesh.v.shape[0]))]


def getposition(x):
    return x['position']


def getnormal(x):
    return x['normal']


def getcolor(x):
    return x['color']


st.deck_gl_chart(
    layers=[{
        'id': 'pointCloud',
        'radiusPixels': 1,
        'type': 'PointCloudLayer',
        'data': data,
        'getPosition': getposition,
        'getNormal': getnormal,
        'getColor': getcolor,
    }])

And this is the error I’m getting:

Do you have any idea?
I don’t have to make it work, if there are easier ways to plot point clouds or, even better, triangular meshes

Thanks!

edit.

Here another working example, in javascript: https://deck.gl/#/examples/core-layers/point-cloud-layer
Code here: https://github.com/uber/deck.gl/tree/7.3-release/examples/website/point-cloud

From what I understood, I should use OrbitView: https://deck.gl/#/documentation/deckgl-api-reference/views/orbit-view

Is it possible to do this in streamlit?

monchier · December 5, 2019, 2:19am

Hi @luca,
thanks for posting. This is a really cool use case! st.deck_gl_chart is a bit of a manual port of DeckGl, so we are planning to replacing it with PyDeck. PyDeck is going to allow us to expose a much better API. This will likely happen by the end of December!

In the meantime, the problem with your code is you’re passing functions into the spec dict, and functions can’t be serialized. So instead what you have to do is create a dataframe with column names x, y, z, normalX, normalY, normalZ, colorR, colorG, colorB and then pass that dataframe into the PointCloudLayer dictionary.

Let me know if this helps!

Thanks,
Matteo

luca · December 5, 2019, 5:04am

Hi @monchier,
Thank you for your reply! It’s great that there will be better API (and by the way, I’m going to check out PyDeck, I didn’t know about that project)

I think I tried to do what you describe, but unfortunately It is not working exactly as expected:


df = pd.DataFrame({
    'x': meshA.v[:, 0],
    'y': meshA.v[:, 1],
    'z': meshA.v[:, 2],
    'normalX': normalsA[:, 0],
    'normalY': normalsA[:, 1],
    'normalZ': normalsA[:, 2],
    'colorR': np.abs(meshA.v[:, 0]),
    'colorG': np.abs(meshA.v[:, 1]),
    'colorB': np.abs(meshA.v[:, 2]),
})
st.deck_gl_chart(
    layers=[{
        'id': 'pointCloud',
        'radiusPixels': 1,
        'type': 'PointCloudLayer',
        'data': df,
    }])

Produces:

Zooming-in:

I think the problem is that I’m not specifying an OrbitView, and it defaults to this one.
Do you know how to specify an orbit view?

monchier · December 5, 2019, 6:20am

I actually don’t know much about the OrbitView. Can you provide meshA so that I can play with your example? What exactly is your expectation here? I would expect i) no map and ii) a 3D rendering. Am I correct?

luca · December 5, 2019, 6:28am

Yes exactly. An interactive plot, where I can rotate the point cloud or zoom-in and zoom-out.

i.e. the live top example here:
https://deck.gl/showcases/gallery/point-cloud-layer

Or here (somehow there are some problems with the link, try to copy-paste it)
https://deck.gl/#/examples/core-layers/point-cloud-layer

I’ll post here the meshA in an hour, just the time to get to the office!

Thanks!

luca · December 5, 2019, 8:43am

@monchier Sorry for the delay.

Here you can find the .pickle file needed to run the code.

This is the code to run a minimal working example:

import streamlit as st
import pickle as pk
import numpy as np
import pandas as pd

st.title('Mesh demo')

with open('mesh.pickle', 'rb') as f:
   mesh = pk.load(f)

vertices = mesh['vert']
normals = mesh['normals']

df = pd.DataFrame({
   'x': vertices[:, 0],
   'y': vertices[:, 1],
   'z': vertices[:, 2],
   'normalX': normals[:, 0],
   'normalY': normals[:, 1],
   'normalZ': normals[:, 2],
   'colorR': np.abs(vertices[:, 0]),
   'colorG': np.abs(vertices[:, 1]),
   'colorB': np.abs(vertices[:, 2]),
})

st.deck_gl_chart(
   layers=[{
       'id': 'pointCloud',
       'radiusPixels': 1,
       'type': 'PointCloudLayer',
       'data': df,
   }])

This is the output I get:

monchier · December 5, 2019, 6:01pm

Hi @luca,
thanks a lot for the info. I am able to reproduce what you see. I think the course of action for this is informing the development of the new PyDeck feature and make it that with the new API it works as expected for this use case. I’ll bring the use case up internally and get back to you.

Matteo

luca · December 5, 2019, 6:25pm

Hi @monchier,

Thank you very much! I’m looking forward to it!

I think that the whole geometry process group at my university could make use of this feature (together with triangular meshes, even though they seems even less supported).

monchier · December 5, 2019, 10:05pm

The new PyDeck API should allow you to render any valid PyDeck object, to say, that, if you are able to write a valid PyDeck object it should work with Streamlit. I assume this should take care of the OrbitView if needed.

For the map, I opened a feature request to make it possible to disable that. it should be easy, once we have the new API.

As for triangular meshes, would you able to get these to work with PyDeck or you need a different library? Let me know, in case, I can open a feature request for those as well, even though it may take longer if it is a brand new library.

More details and tracking:

PR for the new PyDeck API:

Feature request for turning off the map:

luca · December 5, 2019, 10:28pm

The new PyDeck API should allow you to render any valid PyDeck object, to say, that, if you are able to write a valid PyDeck object it should work with Streamlit. I assume this should take care of the OrbitView if needed.

I’m not sure about this.
From what I understood, what is missing is the ability to change the camera behaviour (i.e. the View in deck.gl language).

The same data/object can be displayed in different ways, but the object definition must be rendered with the correct View.

Here there is a brief explanation about the Views:

github.com

visgl/deck.gl/blob/master/docs/developer-guide/views.md

# Views and Projections

The same data can be drawn differently to screen based on what projection method is used. deck.gl's view system defines how one or more cameras should be set up to look at your data objects. The default view used in deck.gl is the [MapView](../api-reference/core/map-view.md), which implements the [Web Mercator projection](https://en.wikipedia.org/wiki/Web_Mercator_projection). 
The view system is designed to be flexible and composable and can handle many different configurations such as side-by-side views, overlapping views etc. If you plan to work with non-geospatial data, or show more than a single standard viewport, it may be worth spending some time to get familiar with the `View` API.

View classes enable applications to specify one or more rectangular viewports and control what should be rendered inside each view.

<table style={{border:0}} align="top">
  <tbody>
    <tr>
      <td style={{verticalAlign:'top'}}>
        <img height="200" src="https://raw.github.com/visgl/deck.gl-data/master/images/docs/minimap.gif" />
        <p><i>A "minimap" app, implemented as two overlapping, partially synchronized MapViews</i></p>
      </td>
      <td style={{verticalAlign:'top'}}>
        <img height="200" src="https://raw.github.com/visgl/deck.gl-data/master/images/docs/first-person-view.gif" />
        <p><i>A vehicle log rendered from the driver's perspective, implemented with FirstPersonView</i></p>
      </td>
      <td style={{verticalAlign:'top'}}>
        <img height="200" src="https://raw.github.com/visgl/deck.gl-data/master/images/docs/orthographic-view.gif" />

This file has been truncated. show original

If the new PyDeck provides the capability to change the View used then it should be ok!

So the issue should be “Allowing the selection of other Views other than the default MapView” (rather than turning off the map, since the map comes from the default view MapView that is being used).

As for triangular meshes, would you able to get these to work with PyDeck or you need a different library? Let me know, in case, I can open a feature request for those as well, even though it may take longer if it is a brand new library

I don’t know very well deck.gl, but it doesn’t seem capable to plot triangular mesh. I may be wrong.

An integration with open3D would be very much appreciated!

http://www.open3d.org/

monchier · December 6, 2019, 7:08pm

Sorry for the delay, but took some time to do some homework. For open3D, I will open a feature request and track it when we discuss it internally.

For PyDeck, I haven’t found support for OrbitView, as you said, but I managed to get a 3D rendering of your dataset (I believe…) with pure PyDeck I had to scale aggressively the z axis.

Please, feel free to correct me if I am off on some technical matter

Pasting the Python script below together with the output I am seeing.

import pydeck as pdk
import pickle as pk
import numpy as np
import pandas as pd

with open('mesh.pickle', 'rb') as f:
   mesh = pk.load(f)

vertices = mesh['vert']
normals = mesh['normals']

df = pd.DataFrame({
   'x': vertices[:, 0],
   'y': vertices[:, 1],
   'z': vertices[:, 2],
   'normalX': normals[:, 0],
   'normalY': normals[:, 1],
   'normalZ': normals[:, 2],
   'colorR': np.abs(vertices[:, 0]),
   'colorG': np.abs(vertices[:, 1]),
   'colorB': np.abs(vertices[:, 2]),
})

point_cloud = pdk.Layer(
    'PointCloudLayer',
    df[['x', 'y', 'z']],
    get_position='[x, y, z * 100000]',
    get_normal=[0, 0, 1],
    pickable=True,
    auto_highlight=True,
    point_size=1)

view_state = pdk.data_utils.compute_view(df[['x', 'y']], 0.9)
view_state.max_pitch = 360
view_state.pitch = 80
view_state.bearing = 120

# Render
r = pdk.Deck(layers=[point_cloud], initial_view_state=view_state, map_style=''
        )
r.to_html('demo.html', notebook_display=False)

monchier · December 6, 2019, 7:11pm

FR for Open3D: https://github.com/streamlit/streamlit/issues/813

luca · December 6, 2019, 8:37pm

Thank you for your time!

For PyDeck, I haven’t found support for OrbitView , as you said, but I managed to get a 3D rendering of your dataset (I believe…) with pure PyDeck I had to scale aggressively the z axis.

Please, feel free to correct me if I am off on some technical matter

Pasting the Python script below together with the output I am seeing.

Yes, I’m seeing the same result!
Thank you very much!

The problem is that the plot is “behaving like a map” (i.e. pan instead of rotate the object)

From your starting point I have been able to make the OrbitView work, however it’s a bit awful to work with (especially the camera).
In particular I’ve played some time with the initial_view_state but I’ve not been able to move the camera such that it correctly renders the points at a reasonable distance. Centering the point cloud and set target=[0, 0, 0] wasn’t enough.

So I resorted to scale the points (stretching them) just to visualize them and make sure that the plot works correctly. It seems to work!

import pydeck as pdk
import pickle as pk
import numpy as np
import pandas as pd
from pydeck import View

with open('mesh.pickle', 'rb') as f:
    mesh = pk.load(f)

vertices = mesh['vert'] * 1000
vertices = vertices - np.mean(vertices, axis=0)
normals = mesh['normals']

df = pd.DataFrame({
    'x': vertices[:, 0],
    'y': vertices[:, 1],
    'z': vertices[:, 2],
    'normalX': normals[:, 0]*255,
    'normalY': normals[:, 1],
    'normalZ': normals[:, 2],
    'colorR': np.random.rand(normals.shape[0]) * 255,
    'colorG': np.random.rand(normals.shape[0]),
    'colorB': np.random.rand(normals.shape[0]),
})

point_cloud = pdk.Layer(
    'PointCloudLayer',
    df[['x', 'y', 'z']],
    get_position='[x, y, z]',
    get_normal=[0, 0, 1],
    pickable=True,
    auto_highlight=True,
    radiusPixels=5,
)

r = pdk.Deck(layers=[point_cloud],
             views=[View(type='OrbitView', controller=True, )],
             initial_view_state={
                 'target': [0, 0, 0],
                 'orbitAxis': 'Z',
                 'rotationX': 5,
                 'rotationOrbit': 5,
                 'zoom': 0})
r.to_html('demo.html', notebook_display=False)

Here what I visualize, that can be freely rotated:

So to recap:

The OrbitView seems to work in PyDeck, thus it should work in the future streamlit. That’s great.
It is super hard to manually center the camera around the points and to correctly set rotationX and rotationOrbit (I suppose all the utility functions are made with MapView in mind, thus not working correctly). Maybe even due to the lack of documentation.
I don’t get why it’s not enough to: (1) center the point cloud (2) set the target, i.e. the center of the world, to [0, 0, 0] (3) set the zoom level appropriately. Doing this it’s not enough to display the point cloud correctly.
The colors aren’t working (maybe it’s missing the light?)

Some sources:

OrbitView class documentation: link
OrbitView sample usage in .js: link

In the wishlist:

Ease the camera adjustment in the OrbitView. I think reasonable defaults could be automatic: center of the world as center of the point cloud, zoom level such that the whole point cloud is displayed

luca · December 15, 2019, 1:46pm

Small update, I investigated a bit plotly.
It’s possible to plot 3D meshes and point clouds with plotly, it works fairly well.

omshinde · June 13, 2020, 4:29pm

Hi!
This thread is superb! Sincere thanks to the authors involved!

To add on top of visualizing point cloud models which has been shown above, I wish to share that this works perfectly for 3D point cloud scenes as well with some modifications. Specifically,

The original point cloud needs to be re-projected to epsg 4326 to convert x, y, z values to latitude, longitude, elevation values.
Scaling is required in all dimensions depending on the x, y, z values.

PFA image for reference.

My query:
I would be highly grateful to know if it could be possible to add color values based on any dataframe column…maybe z values with some classification in ranges.

Thanks!

PS: I tried exploring pydeck.data_utils.color_scales.assign_random_colors() but couldn’t get through.

deckPointCloud

Topic		Replies	Views
3D meshes and streamlit Using Streamlit	1	1013	February 2, 2023
How to Make Point Cloud Visualization Work in Streamlit: Using Streamlit to open a .pcd file and make a 3D visualization with stpyvista Using Streamlit discussion	2	239	February 19, 2025
New Streamlit Release: 0.53.0 Official Announcements release-notes	9	2631	November 19, 2021
Feature request: Support for additional deck.gl layers/pydeck bindings within streamlit Random geospatial	6	2504	November 19, 2021
Deckgl LineLayer Example Using Streamlit	3	1118	November 19, 2021

3D data visualization: point clouds and triangular meshes

Related topics

Hello there 👋🏻

Cookie settings

Strictly necessary cookies

Performance cookies

Functional cookies

Targeting cookies