I’m experiencing a weird behavior with the camera widget when combined with a chat interface.
Observed behavior: photo camera disappears first time photo is taken but subsequent photos never disappear
Case 1:
- Talk in chat
- Take photo (photo disappears, there seems to be a sort of rerun but i don’t have rerun in code)
- Clear photo
- Take photo again (photo shows up and doesn’t disappear)
Case 2:
- Take photo with no chat yet (photo appears fine)
- Talk once in chat (photo stays)
- another chat message (photo disappears, weird ghost camera widget inside chat message)
- Clear photo
- Take photo again (this time it never disappears)
I have no idea how to reproduce or debug this.
This is what I mean by photo disappearing. The camera shows as if a photo is taken but doesn’t show any image
Edit:
Looking at the widget source code I realized that the camera returns an UploadedFile object whenever you take a picture
Returns
-------
None or UploadedFile
The UploadedFile class is a subclass of BytesIO, and therefore
it is "file-like". This means you can pass them anywhere where
a file is expected.
I was wondering if it could be something related to this? Ensure file_uploader doesn't trigger needless reruns by AnOctopus · Pull Request #7641 · streamlit/streamlit · GitHub
Edit 2:
I think that the ghosting behavior of widgets in streamlit is related to this. I added a breakpoint inside the camera_input function to document step by step what is happening. Here is a description: (I put a post it note in front of the camera and that’s why it appears yellow)
First photo: Camera and chat are ready
Second photo: I take a photo by clicking on the Take Photo button and photo is taken and displayed inside camera display as it should. This is evident by the Clear photo text
Third photo: I write a message on the chat. Photo remains in the camera display as it should
Fourth photo: this is where the weird part happens. First I write another message on the chat which makes the app reload including the camera widget. I took a screenshot just after this section of code on the camera_input function
camera_input_state = register_widget(
"camera_input",
camera_input_proto,
user_key=key,
on_change_handler=on_change,
args=args,
kwargs=kwargs,
deserializer=serde.deserialize,
serializer=serde.serialize,
ctx=ctx,
)
self.dg._enqueue("camera_input", camera_input_proto)
As you can see in the pic, there’s a ghosting of the camera with the photo and the expander widget while the model generates the answer. But the photo on the actual camera widget is gone and replaced by a blank screen.
Fifth photo: after the answer from the model comes the camera widget shows up with a blank screen but shows as if a picture has been taken.
Now if I clear the photo and take another and continue chatting the photo stays correctly in the camera display and never disappears so this only happens with the first photo.
Edit 3 (and yet another edit):
So I’m not sure this has to do with how the Deltagenerator works (I still don’t get that completely) but just in case it is relevant, this is the part of my code where I render the camera. As you can see the camera is rendered inside a container (a column) that is inside another container.
def display_chat_interface(self):
"""
Displays the chat interface, manages the display of conversation history, and handles user input.
This method sets up the interface for the chat, including:
- Fetching and displaying the system prompt.
- Updating session states as necessary.
- Displaying conversation history if any.
- Handling user input.
- Displaying uploaded images and URLs if provided.
It also provides an option to view or edit the system prompt through an expander in the layout.
"""
if 'selected_chatbot_path' in session_state and session_state["selected_chatbot_path"]:
description = self._fetch_chatbot_description()
if isinstance(description, str) and description.strip():
# Initialize variables for uploaded content
uploaded_images = session_state.get('uploaded_images', [])
image_urls = session_state.get('image_urls', [])
# Set layout columns based on whether there are images or URLs
if uploaded_images or image_urls:
col1, col2 = st.columns([2, 1], gap="medium")
else:
col1, col2 = st.container(), None
with col1:
# Display chat interface header and model information if applicable
self._display_chat_interface_header()
# Option to view or edit the system prompt
with st.expander(label=session_state['_']("View or edit system prompt"), expanded=False):
self._display_prompt_editor(description)
st.markdown("""---""")
if session_state['activate_camera']:
col3, col4, col5 = col1.columns([1, 2, 1])
col4.camera_input(
session_state['_']("Take a photo"),
on_change=self._store_camera_photo_info,
key='your_photo')
description_to_use = self._get_description_to_use(description)
# Displays the existing conversation history
conversation_history = session_state['conversation_histories'].get(session_state[
'selected_chatbot_path_serialized'],
[])
self._display_conversation(conversation_history, col1)
# If col2 is defined, show uploaded images and URLs
if col2:
with col2:
self._display_images_column(uploaded_images, image_urls)
# Handles the user's input and interaction with the LLM
self._handle_user_input(description_to_use, col1)
else:
st.error(session_state['_']("The System Prompt should be a string and not empty."))
Edit 4: If i display the camera not in col4 but like this in the above code
st.camera_input(
session_state['_']("Take a photo"),
on_change=self._store_camera_photo_info,
key='your_photo')
the problem goes away but I really need the camera inside another container to control the side of the display, otherwise it becomes huge in big screens.
Edit 5: Partial solution. So basically the solution I found is to keep the camera display outside any nested container which is not ideal but a workaround for now. This is how I changed the code:
def display_chat_interface(self):
"""
Displays the chat interface, manages the display of conversation history, and handles user input.
This method sets up the interface for the chat, including:
- Fetching and displaying the system prompt.
- Updating session states as necessary.
- Displaying conversation history if any.
- Handling user input.
- Displaying uploaded images and URLs if provided.
It also provides an option to view or edit the system prompt through an expander in the layout.
"""
if 'selected_chatbot_path' in session_state and session_state["selected_chatbot_path"]:
description = self._fetch_chatbot_description()
if isinstance(description, str) and description.strip():
if session_state['activate_camera']:
col3, col4, col5 = st.columns([1, 2, 1])
col4.camera_input(
session_state['_']("Take a photo"),
on_change=self._store_camera_photo_info,
key='your_photo')
# Initialize variables for uploaded content
uploaded_images = session_state.get('uploaded_images', [])
image_urls = session_state.get('image_urls', [])
# Set layout columns based on whether there are images or URLs
if uploaded_images or image_urls:
col1, col2 = st.columns([2, 1], gap="medium")
else:
col1, col2 = st.container(), None
with col1:
# Display chat interface header and model information if applicable
self._display_chat_interface_header()
# Option to view or edit the system prompt
with st.expander(label=session_state['_']("View or edit system prompt"), expanded=False):
self._display_prompt_editor(description)
st.markdown("""---""")
description_to_use = self._get_description_to_use(description)
# Displays the existing conversation history
conversation_history = session_state['conversation_histories'].get(session_state[
'selected_chatbot_path_serialized'],
[])
self._display_conversation(conversation_history, col1)
# If col2 is defined, show uploaded images and URLs
if col2:
with col2:
self._display_images_column(uploaded_images, image_urls)
# Handles the user's input and interaction with the LLM
self._handle_user_input(description_to_use, col1)
else:
st.error(session_state['_']("The System Prompt should be a string and not empty."))