Huge Log loading makes UI slow

Using Streamlit
1

Hi,
So I made a locally deployed streamlit app for a server response based interaction thingy.
So the server gives HUGE quantities of logs for every response(which I cant help to reduce).
I have to add the LOGS at the bottom of the streamlit app with expander, so that people can use it for debugging. So how I am doing this is, I am saving the log outputted to a FILE, which is then read and displayed. The problem is that it is significantly Slowing down the UI after a few requests, cause the logs get added up. And each time streamlit reruns due to interactions, it is painfully slow.

Here are some codes for the same :

# In utils.py, these 2 below functions are called in streamlit app for log read/write
def log_server():
    """Read from the log file"""
    try:
        with open(LOG_FILE, "r") as f:
            lines = []
            for line in f:
                lines.append(line)
            return "".join(lines)
    except FileNotFoundError:
        return "No log file found yet."

def log_write(proc_name: str, log_message: str):
    """
    Write a message to the log file, in format "[proc_name] log_message".
    proc_name: "Server stderr" or "Streamlit" are standard. You can use any name.
    log_message: The message to log.
    """
    try:
        with open(LOG_FILE, "a") as f:
            f.write(f"[{proc_name}] {log_message}" + "\n")
    except Exception as e:
        print(f"Error writing to log file: {e}")

# In my main streamlit_ui.py file
## LOGS SECTION
if "log_cleaned" not in st.session_state:
    st.session_state.log_cleaned = False

st.subheader("Server Stdout/Stderr", help = "Logs are written to LeanAide-Streamlit-Server log file and new logs are updated after SUBMIT REQUEST button is clicked. You might see old logs as well.")
with st.expander("Click to view Server logs", expanded=False):
    if log_out := log_server():
        height = 500 if len(log_out) > 1000 else 300
        st.text_area(
            "Server Logs",
            value=log_out if not st.session_state.log_cleaned else "",
            height= height,
            placeholder="No logs available yet.",
            disabled=True,
            help="This shows the server logs. It is updated after each request is processed.",
        )

    else:
        st.code("No logs available yet.", language="plaintext")
        
    @st.dialog("Confirm Server Log Cleaning")
    def clean_log():
        """Function to clean the server logs."""
        st.write("Are you sure you want to clean the server logs? This will delete all the logs in the server log file.")
        if st.button("Yes"):
            st.session_state.log_cleaned = True
        if st.button("No"):
            st.session_state.log_cleaned = False

        if st.session_state.log_cleaned:
            try:
                with open(LOG_FILE, "w") as f:
                    f.write("")  # Clear the log file
                st.success("Server logs cleaned successfully! Please UNCHECK THE BOX to avoid cleaning again.")
            except Exception as e:
                st.error(f"Error cleaning server logs: {e}")
        
        st.session_state.log_cleaned = False
    if st.checkbox("Clean Server Logs. Read the help text before checking this box", value=st.session_state.log_cleaned, key="clean_log", help="Check this box to clean the server logs. This will delete all the logs in the server log file."):
        clean_log()

I have a clean_log button, cause at this point it is very necessary.

I was thinking about st.cache_data but I am not sure on how to be use that here, since I already have a LOG_FILE. Anyone has a better way going about this?? How I am sure that Logging is causing the slowness, because every time I clean the log, the UI is fast and beautiful showing its full glory!
Please share.

One more query, how to have the scrollbar for the logs at the bottom by default? Scrolling down is hard, I tried to looking around in other topics and discussions, but none worked.

Thanks in advance!

2

Actually, that proves that logging is not the problem, because the app keeps logging at the same pace after clearing the log. I think it is either reading the file or writing the log to the UI or both.

You can try

  • Keeping the log in memory so that you don’t need to read a (potentially) huge file on each rerun.
  • Show the logs on demand, so they are not written to the UI on each interaction.
3

Yes thanks. This is good enough! It’s much better now!

What about the scroll bar at the bottom query? any ideas on that

4

I guess you would have to inject some javascript. I think there are several posts about this in the forum.

5

As an alternative, I’d reverse the order of the items in the container and show the most recent log entries on top.

image
image1487Ă—912 60.9 KB

import streamlit as st

css = """
.st-key-my-log-div {
    flex-direction: column-reverse;
}
"""
st.html(f"<style>{css}</style>")

"## App"
"*Log*"
with st.container(key="my-log-div", height=250):
    for i in range(100):
        f"`[Log {i:02d}]` This is log entry #{i}"

"More app content " * 10
1 Like
6

Yeah since now I am using a buffer instead of a file. It is much easier to do this. Thanks