Coding AIs are not good at Coding Streamlit

🦄 Random
1

We know that coding AIs can be amazing coders but they hallucinate when coding Streamlit. Here’s an example of a request to code an text_input() callback. It hallucinates that callbacks have a parameter and doesn’t show how to access the new input. Perhaps Streamlit is under represented in training data or were callbacks not originally supported? Results are so much better with other apis such as SQL or Pandas.

I would not expect these kinds of hallucinations if trained on API Reference manuals, programming language manuals or Swagger files. I wonder if these kinds of documents are given priority for training?

Can Streamlit do something about getting better results?

Here’s the problematic response from Bing, ChatGPT and Perplexity (Code Pilot?).

import streamlit as st

def text_input_callback(value):
    st.write("Text input value changed to:", value)

st.write("Enter some text below:")
text_input_value = st.text_input("Text Input", value="", on_change=text_input_callback)
2

Hi @bsalita,

Thanks for posting!

The models are definitely flawed from their training. We don’t have any insights on how how much Streamlit material was part of their training.

We however index more on using our robust documentation, blogs, and this forum as well since it’s ever-evolving with new features we add.

3

Claude 2 was just released. I reran my tests. Claude 2, Bing and ChatGPT 3.5 still give the same wrong answer. GPT 4 also gave the wrong answer using default model but the correct answer using “Code Interpreter” model.

So it’s best to use GPT4 Code Interpreter but it’s currently limited to 25 prompts every 3 hours.