Modifying LLM response

LLMs and AI
1

I’m working on a chatbot with Llamaindex and am using streamlit to build the app. I’m using a basic template found in the streamlit repo (llamaindex-chat-with-streamlit-docs/streamlit_app.py at main · streamlit/llamaindex-chat-with-streamlit-docs · GitHub) and the code displaying the LLM response is shown below:

# If last message is not from assistant, generate a new response
if st.session_state.messages[-1]["role"] != "assistant":
    with st.chat_message("assistant"):
        response_stream = st.session_state.chat_engine.stream_chat(prompt)
        st.write_stream(response_stream.response_gen)
        message = {"role": "assistant", "content": response_stream.response}
        # Add response to message history
        st.session_state.messages.append(message)

This might not be the most efficient method, but I am trying to verify some of the text in the llm response before displaying it to the user. With this method I assume that I can not use the “st.write_stream(response_stream.response_gen)” line since the llm has to finish its entire response before I can verify the text. So, what I want to know is:

  1. I can access the response string using response_stream.response but it returns as empty maybe because it is still generating the answer. How do I obtain the response when it is completed writing the response?
  2. Is there a way to display a loading circle while it generates this response?
    Thanks
2

The complete response could be in:

response_stream.response

You can inspect it before streaming. Try this idea.

if st.session_state.messages[-1]["role"] != "assistant":
    with st.chat_message("assistant"):
        response_stream = st.session_state.chat_engine.stream_chat(prompt)

        # Inspect the response
        complete_response = response_stream.response
        response_ok = verify_response(complete_response)

        # Stream response if ok.
        if response_ok:

            # Show the response in streams.
            st.write_stream(response_stream.response_gen)

            # Format the response 
            message = {"role": "assistant", "content": response_stream.response}

            # Add response to message history
            st.session_state.messages.append(message)
3

Unfortunetly, response_stream.response is still empty with this method and no response is shown when I try out the app. From my tests it seems that you have to run the st.write_stream(response_stream.response_gen) line in order to fill response_stream.response.

So if I do something like:

st.write_stream(response_stream.response_gen)
response_ok = verify_response(complete_response)
if response_ok:
            # Show the response in streams.
            st.write_stream(response_stream.response_gen)
            ...

The app generates the unverified response and then shows the verified response after. I’m not sure why the line response_stream.response_gen is necessary.