Does changing session state actually change my chatbot's LLM settings and output?

🎈 Using Streamlit
1

I am currently testing my app locally. I plan on deploying it to the cloud, but I don’t think that’s relevant. Here is the github repo: GitHub - angoodkind/theo: Chat with my thesis

The code loads the data for my chatbot as follows:

@st.cache_resource(show_spinner=False)
def load_data():
    with st.spinner(text="Loading and indexing the thesis chapters – hang tight! This should take 1-2 minutes."):
        reader = SimpleDirectoryReader(input_dir="./data", recursive=True)
        docs = reader.load_data()
        # print("# of docs: {}".format(len(docs)))
        
        # parameters for the Service Context
        llm = OpenAI(model="gpt-3.5-turbo-instruct", 
                     temperature=st.session_state.llm_temp, 
                     max_tokens=256,
                     top_p=st.session_state.llm_top_p,
                     system_prompt="You are a smart and educated person, and your job is to answer questions about Adam's thesis. Assume that all questions are related to Adam's thesis. Keep your answers based on facts – do not hallucinate features.")
        embed_model = OpenAIEmbedding()
        text_splitter = SentenceSplitter(chunk_size=1024, chunk_overlap=20)
        prompt_helper = PromptHelper(
            context_window=4096,
            num_output=256,
            chunk_overlap_ratio=0.1,
            chunk_size_limit=None,
        )
        # the Service Context is a bundle used for indexing and querying
        service_context = ServiceContext.from_defaults(
            llm=llm,
            embed_model=embed_model,
            text_splitter=text_splitter,
            prompt_helper=prompt_helper,
        )
        
        index = VectorStoreIndex.from_documents(docs, 
                                                service_context=service_context, 
                                                show_progress=True)
        return index

I would like to use sliders to update my LLM’s temperature and top_p parameters. My top_p slider function is set up as:

with st.sidebar:
    lmm_top_p = st.slider(label = "Word Pool Size", key="llm_top_p",
                                min_value=0.0, max_value=1.0, step=.05, value = 0.5,
                                on_change = print_llm_state)

The print_llm_state() function does show that top_p has changed. But I’m wondering if I actually need to call load_data() again to actually update the LLM? Since the LLM is non-deterministic it’s hard to tell if I’m doing this right.

2

You need to call it again, because of temperature and llm_top_p changes.

You need to try:

@st.cache_data

Instead of:

@st.cache_resource

And use a parameter in the function.

@st.cache_data
def load_data(temp, top_p):
    ...

    llm = OpenAI(model="gpt-3.5-turbo-instruct", 
                     temperature=temp, 
                     max_tokens=256,
                     top_p=top_p,
                     system_prompt="You are a smart and educated person, and your job is to answer questions about Adam's thesis. Assume that all questions are related to Adam's thesis. Keep your answers based on facts – do not hallucinate features.")

temp = st.session_state.llm_temp
top_p = st.session_state.llm_top_p
load_data(temp, top_p)

Caching is interesting

3

Ok, I see what you’re saying. How would I. call this from the slider functions, though, where I’m only changing one of those two?

4 
with st.sidebar:
    st.title("How creative?")
    llm_temperature = st.slider(label = "Temperature", key="llm_temp",
                                min_value=0.0, max_value=1.0, step=.05, value = 0.5,
                                on_change = print_llm_state)
    
    lmm_top_p = st.slider(label = "Word Pool Size", key="llm_top_p",
                                min_value=0.0, max_value=1.0, step=.05, value = 0.5,
                                on_change = print_llm_state)

index = load_data()

Change to:

with st.sidebar:
    st.title("How creative?")

    with st.form('form'):
        llm_temperature = st.slider(label = "Temperature", key="llm_temp",
                                min_value=0.0, max_value=1.0, step=.05, value = 0.5,
                                on_change = print_llm_state)
    
        lmm_top_p = st.slider(label = "Word Pool Size", key="llm_top_p",
                                min_value=0.0, max_value=1.0, step=.05, value = 0.5,
                                on_change = print_llm_state)

        submitted = st.form_submit_button("Submit")

    if submitted:
        index = load_data(llm_temperature,  lmm_top_p)

Update load_data

@st.cache_data
def load_data(temp, top):
    with st.spinner(text="Loading and indexing the thesis chapters – hang tight! This should take 1-2 minutes."):
        reader = SimpleDirectoryReader(input_dir="./data", recursive=True)
        docs = reader.load_data()
        # print("# of docs: {}".format(len(docs)))
        
        print("Temperature: {}".format(st.session_state.llm_temp))
        # parameters for the Service Context
        llm = OpenAI(model="gpt-3.5-turbo-instruct", 
                     temperature=temp, 
                     max_tokens=256,
                     top_p=top,
                     system_prompt="You are a smart and educated person, and your job is to answer questions about Adam's thesis. Assume that all questions are related to Adam's thesis. Keep your answers based on facts – do not hallucinate features.")
        embed_model = OpenAIEmbedding()
        text_splitter = SentenceSplitter(chunk_size=1024, chunk_overlap=20)
        prompt_helper = PromptHelper(
            context_window=4096,
            num_output=256,
            chunk_overlap_ratio=0.1,
            chunk_size_limit=None,
        )
        # the Service Context is a bundle used for indexing and querying
        service_context = ServiceContext.from_defaults(
            llm=llm,
            embed_model=embed_model,
            text_splitter=text_splitter,
            prompt_helper=prompt_helper,
        )
        
        index = VectorStoreIndex.from_documents(docs, 
                                                service_context=service_context, 
                                                show_progress=True)
        # print("Index length: {}".format(len(index)))
        return index
5

This raises the error:

StreamlitAPIException: With forms, callbacks can only be defined on the st.form_submit_button. Defining callbacks on other widgets inside a form is not allowed.

Traceback:
File "/Users/adamg/Library/CloudStorage/Dropbox/Programming_Tutorials/Streamlit/theo/streamlit_app.py", line 122, in <module>
    llm_temperature = st.slider(label = "Temperature", key="llm_temp",
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
6

Remove this for both sliders.

on_change = print_llm_state

There are other ways to show the values of user input.