How to avoid sharing of stdout in Streamlit cloud

☁️ Community Cloud
1

Summary

I’m currently using langchain to query some documents and have enabled streaming (per this [documentation](https://python.langchain.com/en/latest/modules/chains/index_examples/chat_vector_db.html#conversationalretrievalchain-with-streaming-to-stdout)). I have written a custom CallbackHandler to get streaming working on Streamlit (to redirect the stdout) When I run the application locally, via multiple terminals, launching the application in multiple browser tabs, it handles concurrent questions perfectly. Each user only sees the streaming of their answer. However, when I deploy this on Streamlit Cloud, I’m having the issue where one user can see the streaming result of another. Streamlit cloud seems to be sharing the stdout. I tried adding stdout to session state, but that didn’t seem to solve the problem. Any ideas on how this can be fixed would be greatly appreciated.

Thanks in advance!

Steps to reproduce

Code snippet:

add code here

If applicable, please provide the steps we should take to reproduce the error or specified behavior.

Expected behavior:

Streaming response (word by word, like ChatGPT) within streamlit

Actual behavior:

Streaming response in terminal.

Debug info

  • Streamlit version: (get it with $ streamlit version)
  • Python version: (get it with $ python --version)
  • Using Conda? PipEnv? PyEnv? Pex?
  • OS version:
  • Browser version:

Requirements file

Using Conda? PipEnv? PyEnv? Pex? Share the contents of your requirements file here.
Not sure what a requirements file is? Check out this doc and add a requirements file to your app.

Links

  • Link to your GitHub repo:
  • Link to your deployed app:

Additional information

If needed, add any other context about the problem here.

2

That should be the problem in my opinion, your test setup is incorrect which does not replicate the actual behavior in a hosted environment, as multiple independent instances are launched here.

1 Like
3

@Franky1 Thanks for spotting that. You’re right. I ran it from one terminal and tried to open the same localhost:port across multiple tabs, and could reproduce the issue locally as well. The root cause seems to be that the stdout is shared and that’s causing the streaming results of one user to be shown to the other. Any tips on how I can resolve this? TIA!

4

Users of an application deployed in Streamlit Cloud cannot see what goes to stdout. Only the owner of the app can see it. You must be misinterpreting whatever is happening.

5

@Goyo Thanks for the response. While you’re right for most applications, as I mentioned in my original post, I’m using this particular application for streaming the GPT responses with Langchain. Per the Langchain documentation, the API streams the response to stdout. Hence the question.

6

It is still true that users cannot see what goes to stdout. So, for users to see the responses, something more must be done. What happens next depends on what exactly that “something more” is.

Anyway, I don’t think you can avoid threads sharing stdout. You can redirect it, but the redirection would affect all threads. You can have several process each with its own stdout, but AFAIK Streamlit sessions are threads in the same process.

If indeed the only way to get a response is having it streamed to stdout, that makes the API almost impossible to use it in threaded code. I find it hard to believe that they took such a design decision.

7

Yes, I’m using a custom class StreamlitOutputCallbackHandler to get the stdout to stream answers inside Streamlit.

8

I think the best answer to your question is a frame challenge: you’d better avoid streaming to stdout.

9

Dear Pruthvi, I am also trying to stream a langchain response with GPT4 using streamlit, but I am struggling at point already. how can I reach out to you?

1 Like
10

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.