Redirecting logger output to a streamlit widget

[david-waterworth]

I have a module which runs a (synchronous) task. It uses the standard python logger, i.e. task.py looks something like

import logging
logger = logging.getLogger(__name__)

def task(args):
    logger.info("Running task")

    logger.info("Finished task")

I want to invoke this task from my app.py and display the log messages as the task runs. I found a few examples that redirected stdout but I cannot get them to work, for example

import logging
logging.basicConfig(level=logging.INFO, stream=sys.stdout)
logger = st.logger.get_logger(__name__)

@contextmanager
def stdout_capture(output_func):
    with StringIO() as stdout, redirect_stdout(stdout):
        old_write = stdout.write

        def new_write(string):
            ret = old_write(string)
            output_func(stdout.getvalue())
            return ret

        stdout.write = new_write
        yield

  output = st.empty()
  with stdout_capture(output.code):
      print("test")
      logger.info("test 2")
      tasks.run()

This correctly test to output.code but not test 2 or messages from the task module, theses log messages are still appearing in my console and not the output widget.

I’ve considered writing my own log handler but I’m wondering if there’s anything obvious I’ve done wrong with my current approach and is this possible / what approaches could I try?

If not I’ll probably use subprocess.Popen and a file log.

[snehankekre]

Hey @david-waterworth

From what you’ve described, it seems like your main objective is to display logs from task.py in your app as they’re generated. What you’ve tried involves redirecting stdout, which is a common strategy for capturing print statements. But it won’t work for capturing log messages directly, as Python’s logging module doesn’t use stdout by default for its output.

Since Streamlit’s loggers don’t write to stdout (the loggers are set up with a StreamHandler that writes to the console) and do not propagate their messages to the root logger (which was configured to redirect to stdout), log messages were not captured by your stdout_capture context manager.

As you’ve suggested, writing your own log handler might be the way to go. You could extend Streamlit’s logger with a custom handler that routes logs to st.empty().code for display, like so:

# app.py
import logging
import task
import streamlit as st
from streamlit.logger import get_logger

class StreamlitLogHandler(logging.Handler):
    def __init__(self, widget_update_func):
        super().__init__()
        self.widget_update_func = widget_update_func

    def emit(self, record):
        msg = self.format(record)
        self.widget_update_func(msg)

logger = get_logger(task.__name__)
handler = StreamlitLogHandler(st.empty().code)
logger.addHandler(handler)

# Run your task
task.run()

# task.py

import logging
logger = logging.getLogger(__name__)
import time
import streamlit as st

def run(*args, **kwargs):
    logger.info("🏃 Running task ")
    time.sleep(2)
    logger.info("🥵 Still running task")
    time.sleep(2)
    logger.info("✅ Finished task")

Breakdown:

• Use get_logger from streamlit.logger, which retrieves a Streamlit-configured logger. It’s already set up with Streamlit’s logging config such as format and level
• Define StreamlitLogHandler – a custom logging handler that overrides the emit method. This method is called whenever a log message is sent to the handler
• In the emit method, the logs are formatted into a message string, and then passed to the widget_update_func, which in this case is st.empty().code
• By adding StreamlitLogHandler to the Streamlit logger (logger.addHandler(handler)), you’re ensuring that any logs processed by this logger are also passed to the custom handler. And since the custom handler is designed to update a Streamlit element, you see the logs in your app

Does this help?

[david-waterworth]

Awesome thanks @snehankekre that works perfectly! I was about to resort to subprocess.Popen (which I’ve done for another process) but don’t have to now.

Not sure if it’s required but I added

logger.handlers.clear()

Before adding the handler to ensure I didn’t get duplicates (I read about this issue in other posts and other solutions are to use a cache decorator around a function to setup the logger, or add the logger to session state).

PS How is the logging module writing to the console without using stdout (or stderr)? I’d always assumed that was the only mechanism to write to the console?

[snehankekre]

Glad it works!

Not sure if it’s required but I added

You’re right. Clearing the log handlers is probably a good idea to avoid duplicates due to script reruns.

PS How is the logging module writing to the console without using stdout (or stderr)? I’d always assumed that was the only mechanism to write to the console?

It’s writing to stderr. The StreamHandler in Python’s logging module sends logs to streams like stderr or stdout or even to files, if configured to do so. When you create a logger and don’t specifically configure its handlers, it uses stderr. This is why you see log messages in the console, even though you’re not explicitly writing to stdout. Here’s where that happens in Streamlit’s logging module:

streamlit/streamlit/blob/28dec526eed312a4aa3e8463c8ba190c584bf9f9/lib/streamlit/logger.py#L66

By default, the StreamHandler in Python logging writes to stderr and not stdout.

This means that even though the log messages are displayed in the console, they are being sent via stderr.

In Streamlit’s case, each logger is configured with this StreamHandler and a specific format, directing its output to stderr. This is why your initial approach with redirecting stdout did not capture these log messages, as they were not being routed through stdout. Additionally, Streamlit loggers do not propagate their log messages to the root logger (logger.propagate = False). This means that any configuration done on the root logger (like redirecting stdout) does not affect Streamlit’s loggers.