CaptionBot- The image captioning bot

Show the Community!
#1

Hi everybody!

CaptionBot takes an image and generates a caption in less than 40 words. It is an implementation of the research paper “Show, Attend and Tell”.
You can check it here. Link for the source code here

Special thanks to Streamlit team, forums and @metasemantic for answering my doubts.

3 Likes
🗞️ Weekly Roundup: Components using Elm, starter kits with Colab, Streamlit + LottieFiles and more!
#2

I gave it a go!

Screen Shot 2020-12-02 at 1.30.13 PM
Screen Shot 2020-12-02 at 1.30.13 PM742×880 120 KB

Looks like my :cat2: is a :dog2: and I never knew! :stuck_out_tongue_winking_eye:

Love the app, very easy to use!

Happy Streamlit-ing!
Marisa

1 Like
#3

Hi! Actually, the model does not work with cats well :sweat_smile: :sweat_smile:. It seems there are no images related to “cats” in Flickr30K dataset. In imageNet dataset also, dogs are more. Something like dogs and people work much better.

Example

Screenshot from 2020-12-03 00-23-52
Screenshot from 2020-12-03 00-23-521920×1080 196 KB

Thanks for trying out!

#4

WHAT! no cats in Flickr30K!?!? :scream_cat: :crying_cat_face:

crazy!

#5

Super cool! I hope this becomes standard, where researcher put up streamlit demos right away.

A few thoughts on the app:

1] Maybe include a link to the publication or model on the app? (Streamlit team, would be really nice if we could modify the footer HINT HINT @Marisa_Smith et al )
2] Auto generate the captions if an image is uploaded … I mean why not?
3] st.file_uploader(label= 'Upload Image', type = ['png', 'jpg', 'jpeg']) why not webp? Dragging an image from another webpage sometimes uses that format and st can handle it just fine.

1 Like
#6

@metasemantic met hahaha will pass along your very subtle suggestion! :crazy_face: :rofl:

2 Likes
#7

@metasemantic psst. add your text in footer like this till then :wink:

import streamlit as st

st.markdown(
    """
    <style>
    footer:after {
        content: " References: XYZ, ABC!" !important;
        color: red !important;
        font-size: 16px !important;
    }
    </style>
    """,
    unsafe_allow_html=True
)

It will look like this,
image

Hope it helps!

1 Like
#8

Thanks again!

  1. The link to the Github repository where I have explained everything is given in the sidebar. But I will add a footer.
  2. I am using a button because I feel it creates some excitement. Also, you can press repeatedly press the button to generate captions. In the earlier version of CaptionBot(trained on Flickr8k and had some mistakes), each button press created new captions. Although, this model is mostly free of mistakes and thus, has less “randomness” in predictions. But still, it might create different captions for complex images.
  3. Thanks. I will add this right now. I was unaware of ‘webp’. :rofl: