How do I create multi-threading in streamlit

Using Streamlit
1

I am building a chatbot application using the Streamlit framework with Python 3.x on my local machine (Windows 11). I have a functionality to measure the LLM accuracy which takes some time, so I want to run a thread to perform the LLM accuracy function separately from the main thread. I tried threading, async functions, and multi-tasking options, but nothing worked. Can anyone help me make it work?

2

Before implementing the LLM accuracy measuring function in the background, it’s important to determine whether the task is CPU-bound (involving intensive local computation) or I/O-bound (involving tasks like API requests or file I/O).

  • For I/O-bound tasks, using multi-threading can be a good option, as Python’s Global Interpreter Lock (GIL) won’t significantly hinder the performance for tasks that are waiting on external resources (e.g., making API calls or reading from disk). In this case, you can safely use threading or asyncio to run tasks concurrently.

  • For CPU-bound tasks, however, the GIL limits the ability of threads to run simultaneously on multiple CPU cores. In this case, using multiprocessing is a better choice, as it allows tasks to run in separate processes, bypassing the GIL and utilizing multiple CPU cores for parallel processing.

Since you mentioned that you’ve already tried threading, async functions, and multi-tasking options without success, I would recommend considering multiprocessing for CPU-bound tasks and ensuring that you’re not facing blocking operations for I/O-bound tasks.

Let me know if you need further clarification or assistance with implementing this!

3

Thanks! I was able to resolve the issue with multiprocessing. Below is the code snippet which is working for me.

# create the pipe

        parent_conn, child_conn = Pipe()
        p = multiprocessing.Process(target=childprocess, args=(user_input,response,child_conn,))
        # jobs.append(p)
        p.start()
        # wait for the return value
        result_ = parent_conn.recv()
        # p.join()