Ignoring event from non-current ScriptRunner: ScriptRunnerEvent.ENQUEUE_FORWARD_MSG

Hi dears! I have an issue, let’s start from mandatory answers and then I will explain context further.

1. Are you running your app locally or is it deployed?
Both ways. On local - no problems, problem appears when I use deployed app
2. If your app is deployed:
a. Is it deployed on Community Cloud or another hosting platform?
Google Kubernetes Engine
b. Share the link to the public deployed app.
It’s private
3. Share the link to your app’s public GitHub repository (including a requirements file).
Its private, but I’ll send you a code.
4. Share the full text of the error message (not a screenshot).

DEBUG:urllib3.connectionpool:http://host:port "POST /route HTTP/1.1" 200 279
INFO:main:LLM answer: ANSWER
2024-02-07 09:50:07.180 Removing orphaned files...
2024-02-07 09:50:07.181 Ignoring event from non-current ScriptRunner: ScriptRunnerEvent.ENQUEUE_FORWARD_MSG
2024-02-07 09:50:07.271 Ignoring event from non-current ScriptRunner: ScriptRunnerEvent.SCRIPT_STOPPED_WITH_SUCCESS
2024-02-07 09:50:07.272 Ignoring event from non-current ScriptRunner: ScriptRunnerEvent.SHUTDOWN
2024-02-07 09:50:30.102 Runtime state: RuntimeState.ONE_OR_MORE_SESSIONS_CONNECTED -> RuntimeState.NO_SESSIONS_CONNECTED

5. Share the Streamlit and Python versions.
streamlit 1.30.0
python 3.10.6


So, the context is:
I running this application

# flake8: noqa
# mypy: ignore-errors
import logging
import os
from typing import Any, Optional
from uuid import uuid4

import streamlit as st
from api_requests import (
    create_chat,
    create_chat_member,
    create_user,
    send_message,
)

# poetry run streamlit run --server.port 8080 --server.address 0.0.0.0 frontend/streamlit_chat_app.py
from dotenv import load_dotenv

# Retrieve API credentials from environment variables
load_dotenv(dotenv_path=".env.local", override=True)

logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

environ = os.environ
session_state = st.session_state


def setup(session_state: Any) -> None:
    if 'session_id' not in st.session_state:
        session_state.session_id = str(uuid4())
    # Initialize session state variables if they don't exist
    if 'chat_id' not in session_state:
        session_state.chat_id = create_chat()

    if 'user_id' not in session_state:
        session_state.user_id = create_user(str(uuid4()))

    # Ensure chat member is created only once per session
    if 'member_created' not in session_state:
        create_chat_member(session_state.chat_id, st.session_state.user_id)
        session_state.member_created = True  # Set a flag to indicate member creation

    session_state.messages = session_state.get('messages', [])


setup(session_state)

st.caption(
    f"User Id: {session_state.user_id}, Chat Id: {session_state.chat_id}",
)

st.title(":car: Digital Pilot")

for message in st.session_state.messages:
    if message["role"] != "system":
        with st.chat_message(message["role"]):
            st.markdown(message["content"])

prompt = st.chat_input("Ask me...")

if prompt:
    user_message = {"role": "user", "content": prompt}
    st.session_state.messages.append(user_message)
    with st.chat_message("user"):
        st.markdown(prompt)

    with st.chat_message("assistant"):
        message_placeholder = st.empty()
        logger.info(f"Going to send LLM prompt: {prompt}")
        try:
            response = send_message(
                prompt,
                session_state.chat_id,
                session_state.user_id,
            )
        except Exception as e:
            logger.error(e)
            message_placeholder.markdown(
                f"Something went wrong. Please try again later. {type(prompt)}",
            )
        else:
            logger.info(f"LLM answer: {response}")
            message_placeholder.markdown(response)  # Display the response

        st.session_state.messages.append(
            {"role": "assistant", "content": response},
        )

Locally and have a nice chat with LLM (thank you for the great simplicity).
But, when I deploy application on K8S, I got the next behavior:
After sending random message to LLM, UI shows for a time that it’s waiting for response, but then - reloads it’s state, and no reply message shows up.
As a result - I have the user message shown, but no answer from the server.
In parallel, DEBUG level streamlit logs shows that streamlit backend receives message from the app backend, but it’s going kinda in old ScriptRunner thread.

Detailed logs of whats going on failure (here two messages wasn’t delivered to frontend):

2024-02-07 09:49:51.687 Beginning script thread
2024-02-07 09:49:51.688 Running script RerunData(widget_states=widgets {
  id: "$$WIDGET_ID-ff3562a5a0f41a6e59e0deb75451175e-None"
  string_trigger_value {
    data: "UserContent1"
  }
}
, page_script_hash='07943488615f26beea8fdf20ec391b09')
2024-02-07 09:49:51.688 Disconnecting files for session with ID aefc9bf0-4d01-48dd-b00e-a62bd9ec0703
2024-02-07 09:49:51.688 Sessions still active: dict_keys([])
2024-02-07 09:49:51.688 Files: 0; Sessions with files: 0
INFO:main:Displaying message: {'role': 'user', 'content': 'Hi!'}
INFO:main:Displaying message: {'role': 'assistant', 'content': 'LLMContent1'}
INFO:main:Sending LLM prompt: UserContent1
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): host:8000
2024-02-07 09:49:59.544 Runtime state: RuntimeState.ONE_OR_MORE_SESSIONS_CONNECTED -> RuntimeState.NO_SESSIONS_CONNECTED
2024-02-07 09:50:00.103 Watcher created for /app/streamlit_chat_app.py
2024-02-07 09:50:00.103 Runtime state: RuntimeState.NO_SESSIONS_CONNECTED -> RuntimeState.ONE_OR_MORE_SESSIONS_CONNECTED
DEBUG:watchdog.observers.inotify_buffer:in-event <InotifyEvent: src_path=b'/app/streamlit_chat_app.py', wd=1, mask=IN_OPEN, cookie=0, name='streamlit_chat_app.py'>
2024-02-07 09:50:00.293 Received the following back message:
rerun_script {
  widget_states {
  }
  page_script_hash: "07943488615f26beea8fdf20ec391b09"
}

2024-02-07 09:50:00.294 Beginning script thread
2024-02-07 09:50:00.295 Running script RerunData(widget_states=, page_script_hash='07943488615f26beea8fdf20ec391b09')
2024-02-07 09:50:00.295 Disconnecting files for session with ID aefc9bf0-4d01-48dd-b00e-a62bd9ec0703
2024-02-07 09:50:00.296 Sessions still active: dict_keys([])
2024-02-07 09:50:00.296 Files: 0; Sessions with files: 0
INFO:main:Displaying message: {'role': 'user', 'content': 'Hi!'}
INFO:main:Displaying message: {'role': 'assistant', 'content': 'LLMContent1'}
INFO:main:Displaying message: {'role': 'user', 'content': 'UserContent1'}
2024-02-07 09:50:00.303 Removing orphaned files...
DEBUG:watchdog.observers.inotify_buffer:in-event <InotifyEvent: src_path=b'/app/api_requests.py', wd=1, mask=IN_OPEN, cookie=0, name='api_requests.py'>
2024-02-07 09:50:00.487 Watcher created for /app/api_requests.py
2024-02-07 09:50:00.489 Script run finished successfully; removing expired entries from MessageCache (max_age=2)
DEBUG:urllib3.connectionpool:http://host:port "POST /route HTTP/1.1" 200 279
INFO:main:LLM answer: ANSWE?
2024-02-07 09:50:07.180 Removing orphaned files...
2024-02-07 09:50:07.181 Ignoring event from non-current ScriptRunner: ScriptRunnerEvent.ENQUEUE_FORWARD_MSG
2024-02-07 09:50:07.271 Ignoring event from non-current ScriptRunner: ScriptRunnerEvent.SCRIPT_STOPPED_WITH_SUCCESS
2024-02-07 09:50:07.272 Ignoring event from non-current ScriptRunner: ScriptRunnerEvent.SHUTDOWN
2024-02-07 09:50:30.102 Runtime state: RuntimeState.ONE_OR_MORE_SESSIONS_CONNECTED -> RuntimeState.NO_SESSIONS_CONNECTED
DEBUG:watchdog.observers.inotify_buffer:in-event <InotifyEvent: src_path=b'/app/streamlit_chat_app.py', wd=1, mask=IN_OPEN, cookie=0, name='streamlit_chat_app.py'>
2024-02-07 09:50:30.692 Watcher created for /app/streamlit_chat_app.py
2024-02-07 09:50:30.692 Runtime state: RuntimeState.NO_SESSIONS_CONNECTED -> RuntimeState.ONE_OR_MORE_SESSIONS_CONNECTED

And how things going on local stage:

digital_pilot-streamlit-app-1  | INFO:__main__:Going to send LLM prompt: Hi!
digital_pilot-streamlit-app-1  | DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): digital_pilot-app-1:8090
digital_pilot-streamlit-app-1  | DEBUG:urllib3.connectionpool:http://digital_pilot-app-1:8090 "POST /chats/65c49d340ad777a0b1675b78/messages HTTP/1.1" 200 237
digital_pilot-streamlit-app-1  | INFO:__main__:LLM answer: Answer
digital_pilot-streamlit-app-1  | 2024-02-08 09:22:12.656 Removing orphaned files...
digital_pilot-streamlit-app-1  | 2024-02-08 09:22:12.705 Script run finished successfully; removing expired entries from MessageCache (max_age=2)

As I can see from logs, remote app have some iterative (ReRun script launches, and local - do not have. And each that ReRun makes all previous requests obsolete)

The one reason I can suggest for know is that K8S running healthchecks, and local env do not - so that healthcheck re-runs script, but why does it affects the sessions of real users? This is not the proper way to run Web App. I think, I don’t understand something here, but I wrote code just like in guides. Will this ignoring happen if, for example, two users will interact with streamlit backend?

Finally: do you have Ideas, how to fix it?

Thank you!

Hello @Rut_Pizhikov,

Here are some considerations:

  1. Ensure that your Kubernetes setup is configured to use sticky sessions if possible.In a Kubernetes deployment, if your application is behind a load balancer without sticky sessions enabled, requests from the same user could be routed to different pods, leading to a loss of session state.

  2. Investigate the Kubernetes pod logs and events to see if there are any restarts or liveness probe failures that could cause unexpected behavior.

  3. Heavy computations or API calls should be cached using st.cache or only triggered by specific user actions rather than running on every rerun.

  4. Streamlit has a server.headless configuration option that can be set to true in your .streamlit/config.toml file for deployments in headless environments like Docker containers in Kubernetes. This ensures Streamlit is optimized for such environments, although it’s more related to how Streamlit is displayed rather than session management.

Hope this helps!

Kind Regards,
Sahir Maharaj
Data Scientist | AI Engineer

P.S. Lets connect on LinkedIn!

➤ Want me to build your solution? Lets chat about how I can assist!
➤ Join my Medium community of 30k readers! Sharing my knowledge about data science and AI
➤ Website: https://sahirmaharaj.com
➤ Email: sahir@sahirmaharaj.com
➤ 100+ FREE Power BI Themes: Download Now

3 Likes