Deploying streamlit on Kubernetes

Deployment
,
1

I’ve deployed a streamlit app to a Kubernetes cluster. I’m using the default config (no server.enableCORS or server.port or browser.serverAdress tweaks as mentioned in other threads here). I’m using a load balancer that wires requests from https://my-streamlit-app.my-domain.com/ (hence port 80) to the container on its port 8501. I use https, so the websocket calls use the wss protocol.

Using port-forwarding, I can confirm the deployment is working. Therefore if I do kubectl port-forward my-pod 5000:8501 and go to http://localhost:5000 everything works as expected.

But if I go to https://my-streamlit-app.my-domain.com/, I see the unfriendly Please wait... and the app doesn’t load.

By looking into the chrome dev tools’ network tab, I see that the websocket call to Request URL: wss://tagging-stg.onefootball.com/stream fails with :

  • either the error message WebSocket is closed before the connection is established.
  • or the message Error during WebSocket handshake: Unexpected response code: 400.

I sense this might be a reverse-proxy/server configuration issue. I’ll leave here the content of the headers for those requests (among other things, but I have no clue if the rest is relevant or not):

Connection: Upgrade
Host: my-streamlit-app.my-domain.com
Origin: https://my-streamlit-app.my-domain.com
Upgrade: websocket

I’m at a loss here. Kubernetes is our deployment of choice at my company and I cannot figure out how to deploy streamlit there so it’s blocking our adoption.

  1. Has anyone seen those websocket errors before?
  2. Is this an issue with tornado or this an issue with the reverse-proxy config ?
  3. Why is the app working fine when port-forwarding and not when accessing the website directly ?

Thanks in advance :pray:

2

I should also mention requests to /healthz have a 200 status. And that I’ve read all posts I could find on this forum including K8 - Error during WebSocket handshake: Unexpected response code: 200

3

Is there any chance you can share your service and deployment definitions?

1 Like
4

Can you share your load balancer/ingress configuration?
Sometimes it helps look at exactly which addresses are requested in the Chrome Network tools. I remember spotting bugs in regards of which addresses are generated by streamlit. Not sure this is the case.

1 Like
5

I can’t really share a reproducible service and deployment definitions as there are a lot of parts that are not directly related to this issue :frowning: .

In order to further debug this issue though, I coded a minimal socket.io app (https://github.com/louisguitton/postit-dashboard, see it live here) and deployed it on my k8s cluster and I could confirm that I’m facing the same issue.
So this is not an issue related to streamlit directly but rather to my load balancer not letting the websocket request complete.

  • On the Kubernetes deployment, it fails with WebSocket connection to 'wss://my-streamlit-app.my-domain.com/stream/?EIO=3&transport=websocket&sid=_UgI94ruB37k3QCNAAAE' failed: Error during WebSocket handshake: Unexpected response code: 400.
  • On the Heroku deployment, the websocket request succeeds with HTTP code 101.
    I’ll look more into this.

Here is the relevant section of the helm chart I’m using currently (and that is failing for now):

apiVersion: v1
kind: Service
metadata:
  name: my-streamlit-app
  namespace: my-namespace
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v0
      kind: Mapping
      name: my-streamlit-app
      host: my-streamlit-app.my-domain.com
      ambassador_id: private
      timeout_ms: 30000
      prefix: /
      service: my-streamlit-app.my-namespace
      ---
      apiVersion: ambassador/v0
      kind: Mapping
      name: my-streamlit-app-websocket
      host: my-streamlit-app.my-domain.com
      ambassador_id: private
      timeout_ms: 9000000000000
      prefix: /stream/
      rewrite: /stream/
      service: my-streamlit-app.my-namespace
      use_websocket: True
7

Hi @louisguitton, did you get any luck figuring out this issue? I am having the exact same issue.
The error is WebSocket is closed before the connection is established.

8

Hi @Freeman_Chen,
I found out that the reverse proxy version installed on the cluster I was trying to deploy to was outdated and did not support websockets. (If like me you’re using ambassador, make sure you’re using 0.50 or later).
Also, the Elastic Load Balancer configs we had did not fit the websocket scenario which requires the backen protocol TCP rather than HTTP.

Ultimately I dropped streamlit and rewrote my app with a different framework, not relying on websockets. (Disclaimer: I was also missing another key feature from streamlit not related to websockets - url routes you can deeplink to - which nudged me in that direction too).

I wish I had seen earlier that streamlit was relying on websockets because this all investigation - although really interesting - was a time sink.

Good luck for your use case