Is it possible to send CURL request to the server?

🎈 Using Streamlit
1

I can do
streamlit run app.py and I can interact with browser. Is it possible to get the result back through CURL request ?

Thanks

2

Hi @Md_Asadul_Islam

Streamlit uses two protocols to communicate between the browser and the server: HTTP and websocket.

And it has 2 main endpoints:

  • / - serves index.html
  • /stream - websocket endpoint that serves the content of your app and receives commands from the app.

If you want to see what’s happening in /stream you need to use a websocket-specific curl command, like below:

curl --include \
     --no-buffer \
     --header "Connection: Upgrade" \
     --header "Upgrade: websocket" \
     --header "Host: localhost:8501" \
     --header "Origin: http://localhost:8501" \
     --header "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" \
     --header "Sec-WebSocket-Version: 13" \
     --output streamlit_websocket_dump.bin \
     http://localhost:8501/stream

However, if you try the above, you’ll see that the streamlit_websocket_dump.bin file is empty. This is because Streamlit clients need to tell the server to rerun the app by sending a special “run the app” message via the websocket.

This is quite annoying to do by hand, but luckily there’s a simple hack around it: just restart your Streamlit server and make sure no browser ever connects to it. Then, run the curl command above.

This works because we treat the very first connection to a Streamlit server differently from others. The first connection immediately receives data from the app without need to send the “run the app” message.

The best way to make sure no browser ever connects to your server is by closing all Streamlit tabs in your browser and adding this to your ~/.streamlit/config.toml:

[server]
headless = true

This will cause Streamlit to not open a browser when you restart the Streamlit server.

Now, if you go through all of the above and then look at a successful capture of streamlit_websocket_dump.bin, you’ll notice it’s an unreadable binary file. That’s because Streamlit uses a binary format for communications: Protocol Buffers (aka “protobufs”)

If you want to decode that format you’ll need to input our protobuf schema into a protobuf decoder tool.

3

Thanks @thiago for details description. Will you please share more details about how to decode binary message using protobuf schema? Possibly an example would be great!

4

The basic steps are:

  1. Install the Protocol Buffer compiler
    # Mac
brew install protoc

# Linux (Ubuntu)
sudo apt-get install protobuf-compiler
  2. Grab our repo, because the schema is there:
    git clone https://github.com/streamlit/streamlit.git
  3. Go into the proto folder:
    cd streamlit/proto
  4. Deserialize some binary protobuf:
    protoc --decode ForwardMsg streamlit/proto/ForwardMsg.proto < file_to_deserialize.bin
    Where ForwardMsg is the schema for messages sent from the server to the client, and BackMsg is the schema for messages sent the other way around.

The problem is that the curl command from the previous post gives you a single file with a bunch of websocket messages one after the other. And it’s hard to tell where one ends and the next one starts.

There must be some off-the-shelf tool you can use for that, but I’m not really familiar with any.

So before going down that route I might as well ask: what are you trying to do? Maybe there’s a solution that doesn’t require all of this?

5

Yes you are right. There are scenarios where this will not be necessary at all. I am trying to put it in production and setting up HTTPS connections from cient to host(The model run by streamlit). I know I can use Flask or any other tool for that purpose. Also I am using a Transformer type pretty heavy model in production.

Thanks