My chat app's chat engine is condense_question but answering everything

[Hai_Linh_Nguyen]

Hello,
I set my chat engine is as below: chat_engine = index.as_chat_engine(chat_mode=“condense_question”, verbose=True)
On my local machine it doesn’t answer question not related to the uploaded file as expected, but after deployment, it answers everything.
My chatapp is at leonatezchat.streamlit.app.
Thank you so much!

[Caroline]

Hey @Hai_Linh_Nguyen,

Can you please share a code snippet so we can try to reproduce the issue you’re seeing?

[Hai_Linh_Nguyen]

sure Caroline! Please find it here:

import streamlit as st
from llama_index import VectorStoreIndex, ServiceContext, Document
from llama_index.llms import OpenAI
import openai
from llama_index import SimpleDirectoryReader
import os
import random
import string

# Create a directory if it doesn't exist
def generate_random_string(length):
    characters = string.ascii_letters + string.digits + string.punctuation
    random_string = ''.join(random.choice(characters) for _ in range(length))
    return random_string
if 'data_dir' not in st.session_state:
    st.session_state['data_dir'] = "./data"+ generate_random_string(10)
    data_dir = st.session_state['data_dir']
    os.makedirs(data_dir, exist_ok=True)
    print('Created dir', st.session_state['data_dir'])

data_dir = st.session_state['data_dir']


# icon list here: https://raw.githubusercontent.com/omnidan/node-emoji/master/lib/emoji.json
st.set_page_config(page_title="Your own chatbot", page_icon="🥷", layout="centered", initial_sidebar_state="auto", menu_items=None)
openai.api_key = st.secrets.openai_key
st.title("Chat with your own chatbot")
industry = st.text_input('Tell me briefly about your product. For example: iPhone - A cutting edge phone with touch screen')
systemprompt = f"You are an expert on {industry} and your job is to answer all questions related to {industry}. Assume that all questions are related to the {industry} only. Keep your answers focus and based on facts, try to convince the users to buy or use the {industry} – do not hallucinate features."

#file uploader
uploaded_file = st.file_uploader('Upload a document')
# Check if a file was uploaded
if uploaded_file is not None:
    file_path = os.path.join(data_dir, uploaded_file.name)
    with open(file_path, "wb") as file:
        file.write(uploaded_file.getbuffer())
    st.write("File saved successfully.")
    

    #Initiate conversation         
    if "messages" not in st.session_state.keys(): # Initialize the chat messages history
        st.session_state.messages = [
            {"role": "assistant", "content": "Ask me anything about the document you just uploaded"}
        ]

    @st.cache_resource(show_spinner=False)
    def load_data():
        with st.spinner(text="Wait a few seconds, I am learning your knowledge ⏱️"):
            reader = SimpleDirectoryReader(input_dir=data_dir, recursive=True)
            docs = reader.load_data()
            service_context = ServiceContext.from_defaults(llm=OpenAI(model="gpt-3.5-turbo", temperature=0.5, system_prompt=systemprompt))
            index = VectorStoreIndex.from_documents(docs, service_context=service_context)
            return index

    index = load_data()
    #chatmode = 'condense_question' 'reAct_agent' 'OpenAI_agent'
    
    chat_engine = index.as_chat_engine(chat_mode="condense_question", verbose=True)

    if prompt := st.chat_input("Your question"): # Prompt for user input and save to chat history
        st.session_state.messages.append({"role": "user", "content": prompt})

    for message in st.session_state.messages: # Display the prior chat messages
        with st.chat_message(message["role"]):
            st.write(message["content"])

    # If last message is not from assistant, generate a new response
    if st.session_state.messages[-1]["role"] != "assistant":
        with st.chat_message("assistant"):
            with st.spinner("Good question, let me think ..."):
                response = chat_engine.chat(prompt)
                st.write(response.response)
                message = {"role": "assistant", "content": response.response}
                st.session_state.messages.append(message)

[Caroline]

Hey @Hai_Linh_Nguyen,

When you say it answers everything, does it return a limited response to those questions that aren’t specific to the knowledge base, or is the answer pretty accurate?

[Hai_Linh_Nguyen]

Hey @Caroline , thank you so much for your quick response!
Today I tried with another context and now it returned correctly, however it still answer other general questions which not mentioned in the given context like: where is New York located (Answer: it’s located in US), What is Klarna (Answer: a financial company which provides services like BNPL, installments, […] ), What is BNPL (Answer: Buy Now Pay Later). The answers are correct, but usually shorter than answers of the same questions on native ChatGPT.
I want it would not answer those questions at all. It works correctly on my local machine but not cloud. I have set chat mode is condense question as you can see but it still answer other questions.

[Caroline]

Hey @Hai_Linh_Nguyen,

Super interesting that it’s working differently on your local machine vs. in Community Cloud – is the code for the app exactly the same in both places?

[Hai_Linh_Nguyen]

Maybe it’s ChatGPT issue not streamlit but it’s interesting. I just did it again: Go to github repo which was used to deploy on Community cloud, download the code to local machine and run it. The result is still strange: on local it doesn’t answer out-of-context questions, but on Community cloud it does.

My github repo is here: https://github.com/leonatez/chatbot/blob/main/chatbot.py