Is it currently possible to select features or draw a selection box on a plot canvas?

Hi all.

Iā€™m trying to build an app that requires that the user either selects an area (bounding box/ rectangle) or specific countries on a map/ colorpleth plot.

Is it currently possible to do this in streamlit? I.e.: can I return the selection box a user drew into a canvas and can I record if a certain polygone was clicked?

Cheers,
C

@cwerner are you building an image annotation pipeline? As someone thatā€™s building their own, I would love a selectable canvas.

Currently Iā€™m on a capped free license on label box.

Labelbox provides polygon draw and box draw:

What if we had:

obj_list = st.canvas_select(img, box = True, polygon = True)

Im pretty sure there is even nicer abstraction that can handle a wider variety of interactions. Imagine, for example something like:

geometry_list = st.geom_canvas(img, box = True, polygon = True, point = True, radius = True)

or imagine if hover state was bound to the geometry object. Such that a hover event updated the geometry list contents. It would be hella expensive processing-wise without carefull/masterful cacheing.

1 Like

Hello everyone,

I also have a mini side project similar to Googleā€™s Quickdraw where I interact with a canvas to draw doodles, export the result as an array to send through a TensorflowJS model, and recognize the drawing (you can test here :wink:).

It would be much faster if instead of pulling a JS project next to my Python TF pipeline, I could just use Streamlit to load the TF model and draw on a canvas which returns numpy arrays. I think if you load a canvas with your image as background we can solve our use cases.

I havenā€™t tested it yet, have you looked at ipycanvas ? Does the API and examples sound nice to you ? Maybe we could have a similar one :

canvas = st.canvas(size=(200, 200))

bg = load_image('test.png')
canvas.draw_image(bg, 50, 50)

canvas.fill_rect(0, 0, 50, 50)

def handle_mouse_down(x, y):
    # Do something else
    pass
canvas.on_mouse_down(handle_mouse_down)

And then eventually it becomes itā€™s own input and we can interact with the polygons ?
ā€¦
Now I see thereā€™s already an issue on this and I have to agree that it would be pretty hard to maintain a new JS dependency only for one streamlit method except if more people need this for their DL pipelines (there is no Canvas component in the frontend library used by Streamlit) so the foreseen way would be to wait for a plugin architecture to push our own plugin, and then merge it into the project if more people see it as a necessary API.

1 Like

This sounds good, with the exception of the mouse_down.

With the mouse_down event being sent, any change in state essentially re-runs the entire stream lit app.

The completion of a box draw or the completion of a polygon draw(once the draw path reconnects with the the first point in the path) happens less frequently so it might be easier on streamlit. I say this because I already see weirdnesses in interactivity as I slide a range slider left and right and the app re-computes.

If we can think of an elegant way to not broadcast every mouse-movement to the streamlit engine, that can make the interface seam more fluid.

Having pre-defined shapes might fit the use case. For your doodling only broadcasting the path between a mouse down and mouse up as one object. Rather than have a coder implement that functionality on their own end.

1 Like

Totally agreeing with you there. On the interactivity side when interacting with the slider, Iā€™ve already created an issue about this, and hopefully the solution can be copied to this problem.

Do you mind putting your thoughts on the existing drawing issue ? Maybe the issue with callbacks over JS mouse events can help steer the conversation around architecturing the plugin system, to give the ability to choose when to rerun the script so everything runs smoothly.

For inspiration on letting a user draw there are a lot of examples in the streaming section of https://holoviews.org/reference/index.html. You can use them via a bokeh or panel app.

If you are dedicated enough and can do some js development you can actually use those examples in streamlit as they are built on top of bokeh.

The key is to find something that can send the selection from front end to backend.

The ā€œCustom widgets hack - loginā€ in the Gallery at awesome-streamlit.org is that key.

You can also learn how widgets are implemented in streamlit and then contribute one.

1 Like

Drawable Canvas should solve most of this issue. Donā€™t hesitate to bring newer use cases directly on the repo for the project ;).

Fanilo

1 Like

Oh, neato. Have to give this a whirl. Would that work with Geopandas/ matplotlib/ cartopy or other geo-plots? I.e. select a set of countries from a map? And show another more detailed analysis for them?