Crosstalk between streamlit sessions with multiple users

Hi - I coded up a cool streamlit app and am going to share it internally. It is an image viewer and has a sidebar with some slider bars. I have this hosted on an internal server with the “streamlit run myapp.py” command.

When multiple users are in the app, things get weird in the sidebar. The sidebar has a few slider bars, a selectbox, a checkbox and a button. I noticed that when someone else views the app in their browser when I have my browser open it seems to me like the sessions are cross talking. On my view I get repeated sliders, repeated buttons, that kind of thing. It seems I can also affect the other users browser by clicking buttons in mine.
Any ideas what could be causing it? When the other user closes their browser then all the weird behavior disappears and the app acts normally. What is the streamlit server model, are we using the same session cookie somehow? It seems like our sessions are sharing state between them.

Here is my code if you want to have a look:

def main():

    file_dict = None
    with open('file_data.json') as f:
        file_dict = json.load(f)

    st.title('Burrito Bounty')
    obj = Image.open('bounty.jpg')
    st.sidebar.image(obj)
    with open('instructions.md', 'r') as file:
        readme_txt = file.read()
    st.markdown(readme_txt)

    keys = list(file_dict.keys())
    index = st.sidebar.slider('Image Index', 0, len(keys) - 1, 0)

    show_fullres = st.sidebar.checkbox('Show Full Res Image')
    load_option = st.sidebar.radio('Select Image To Load',
                                   ('Original RGB', 'Synthetic1', 'Synthetic2', 'Pop Green', 'NRG', 'NIR'))
    adjust_contrast = st.sidebar.checkbox('Contrast Stretch')
    use_column_width = not show_fullres

    image_list = []
    image_types = ['RGB', 'SYNTH', 'NRG', 'NIR', 'SYNTH2', 'POP_GREEN']
    for i in image_types:
        obj = Image.open(file_dict[keys[index]][i])
        image_list.append(np.array(obj))

    if adjust_contrast:
        hi = st.sidebar.slider('Hi Cutoff %', 98.0, 100.0, 99.9, 0.1)
        lo = st.sidebar.slider('Lo Cutoff %', 0.0, 2.0, 0.5, 0.1)
        for idx, image in enumerate(image_list):
            image = contrast_stretch(image, lo=lo, hi=hi)
            image_list[idx] = image

    button = st.sidebar.button('Release Balloons!')
    if button:
        st.balloons()

    if load_option == 'Original RGB':
        st.image(image_list[0], caption='Original Image, Image Index: {}'.format(index), use_column_width=use_column_width)
    elif load_option == 'Synthetic1':
        st.image(image_list[1], caption='Synthetic Image, Image Index: {}'.format(index),
                 use_column_width=use_column_width)
    elif load_option == 'NRG':
        st.image(image_list[2], caption='False Color Infrared Image, Image Index: {}'.format(index),
                 use_column_width=use_column_width)
    elif load_option == 'NIR':
        st.image(image_list[3], caption='NIR Image, Image Index: {}'.format(index),
                 use_column_width=use_column_width)
    elif load_option == 'Synthetic2':
        st.image(image_list[4], caption='Synthetic 2 Image, Image Index: {}'.format(index),
                 use_column_width=use_column_width)
    elif load_option == 'Pop Green':
        st.image(image_list[5], caption='Pop Green Image, Image Index: {}'.format(index),
                 use_column_width=use_column_width)


if __name__ == "__main__":
    main()
2 Likes

Hi @cpadwick

This sounds quite alarming.

Our entire architecture is such that each connected user has her own session object in the server, and her own separate thread where the app’s source file is executed. While the source file executes, the Streamlit library in that thread writes can only write to that specific session object (because that’s the only session it even has a reference to).

Then, the Streamlit server periodically loops through its websocket-to-session dict and writes any outstanding messages from each session into its corresponding websocket.

So if this is indeed a bug, it would have to be particularly nasty one that only occurs in very specific conditions I can’t begin to imagine, and which we haven’t encountered yet. That said, I’m very intrigued by this and would love to get to the bottom of it!

Since I can’t reproduce this bug using any Streamlit app in my arsenal, and since your app doesn’t appear to be doing anything crazy, would you mind either:

  1. Providing the JSON and image files that will allow me to run your app, above

or

  1. Finding a minimal example app where this bug reproduces

?

@cpadwich, have you solved this problem? what was the issue?

@thiago Hello! We’re experiencing this issue when running on Google Compute Engine, running in a Docker container. It happens to be behind an identity-aware proxy. What’s the websocket-to-session object keyed on? An effect of the proxy is that all users appear to be visiting from the same IP.

It seems to happen even on a vanilla hello-world example on this infra.

1 Like

I also get cross-talk. I have streamlit running inside docker and kubernetes. The client is running in multiple tabs on the same machine.

1 Like

I think I know what could be causing this. Will investigate tomorrow.

2 Likes

The tests took longer than I predicted, but the PR is ready now. Just waiting for a code review!

3 Likes

I am experiencing crosstalk using the file_upload widget. Any chance the same bug is there?

Hey @ifeherva :wave:

What Streamlit version are you on? The fix for this was pushed on 0.54.0 which was released yesterday. If on an earlier version, mind upgrading to 0.54.0 to see if this helps?

I am on 0.54.0.

The problem can be reproduced using 2 sessions on the same host and with this simple code:

https://raw.githubusercontent.com/MarcSkovMadsen/awesome-streamlit/master/gallery/file_uploader_multiple_files/file_uploader_multiple_files.py

I filed a bug on this issue: https://github.com/streamlit/streamlit/issues/1096

@andy So we originally thought we had crosstalk, but realized that what we were seeing was behavior due to how our load balancer was configured. We are running streamlit in a docker container on GKE behind a load balancer. Our websockets were timing out at 30s by default even if they were active (https://cloud.google.com/load-balancing/docs/backend-service#backend_service_settings). The websocket connection breaking and instantly reconnecting in a browser caused some really odd behavior (multiple plots to load, sometimes certain plots would not load).

Thought it was worth posting - with the load balancer/proxy set to keep alive websocket connections we haven’t seen any issues!

2 Likes

This is awesome. We’d been seeing “random” page refreshes. Changing the session timeout to a longer period of time has made this stop. Thank you for posting!

(That said, I’m not sure that I see a way to “keep alive” indefinitely - we just set a much-longer-than-30s time. Is there a setting I’m missing?)

Hey - totally sorry I missed this reply. We did the same thing, just set the session timeout to the highest value (like 24hours if I recall correctly). Glad this helped!

Hi,

Thanks for the tip. Are we talking about websocket timeouts:

websocket_connect_timeout ######;
websocket_send_timeout #####;
websocket_read_timeout #####;

Or the proxy timeouts:

proxy_connect_timeout 7d;
proxy_send_timeout 7d;
proxy_read_timeout 7d;

Sorry, I just wanted to be sure which one(s) is/are the correct settings especially in Nginx with multiple upstreams.

@maziyarpanahi I am not sure about the nginx config since I am deploying with on GKE with a loadbalancer - the setting for me is just called timeout and defaults to 30s, and defines the time that a websocket can remain open (https://cloud.google.com/load-balancing/docs/backend-service#timeout-setting).

For nginx config my guess is that you have to set the websocket_*_timeout for the websockets to keep alive for longer than their default.

2 Likes

Please be more specific about “periodically”, preferably with a pointer to the source code.

I’m debugging an issues here where one user can connect while the other one can’t at the same time. Then, later both users can connect again.

Hi @StefanBrand

We empty the message queue every 10ms and dispatch messages to the appropriate browser.

The line of code where we send data to the browser is here:

That said, your issue doesn’t sound like it’s related to the line above. To debug, the first place where I’d look is the app’s CPU usage profile. What you describe sounds a lot like the app is somehow hogging the CPU for long periods, causing Streamlit’s Tornado server to starve, making it unable respond to requests.

1 Like