Adding in streamlit chat_input the possibility to add an image

🧩 Custom Components
1

Good morning!

I was thinking to add the possibility to upload an image in chat_input, as OpenAI released the gpt-4-turbo including vision.

That could be great that chat_input could handle text and image inputs.

Thanks!

1 Like
2

Hi @Adlef

Images could be uploaded via st.file_uploader (st.file_uploader - Streamlit Docs)

3

That’s correct, but I was thinking to combine both together, as done by ChatGPT. A drag and drop into the chat_input, instead of combining the prompt written in chat_input and an upload of file. Besides, when sending a prompt via chat, I suppose automatically file_uploader is not cleaned right? It must be handled by code. Or?

4

Unfortunately, st.chat_input accepts text and the only way to get images into the app is via the st.file_uploader. After an image file has been uploaded, you can process it into a format that the OpenAI API can process namely into a NumPy array.

Here’s a code snippet from @Adrien_Treuille_Snow from this post.

import streamlit as st
import numpy as np
from PIL import Image

img_file_buffer = st.file_uploader('Upload a PNG image', type='png')
if img_file_buffer is not None:
    image = Image.open(img_file_buffer)
    img_array = np.array(image)

Hope this helps!

5

Thanks for your answer! I will play with that :slight_smile:

6

I also came across this issue. I think having a component that can send attachments, especially with the vision AIGC models becoming more common, is very important.

I’m working on a custom component with the help of Copilot. It’s now already working but with some caveats. It’s open-sourced here and wish people who have more experience can help improve it! GitHub - AI-Colleagues/st-components

2 Likes
7

Update: I’ve just committed some big improvements. It’s practically usable now. No significant issues, and not very ugly IMO. Here is a video for it. Feel free to submit a PR if you have ideas to improve it further!

1 Like