Streamlit App deployed as Azure WebApp for containers becomes unresponsive over time

Marc · October 9, 2019, 10:00am

Hi Streamlit

I’ve deployed my docker container to Azure WebApps for containers here https://awesome-streamlit.azurewebsites.net/.

It worked fine in the beginning but after some time it has now gotten slow and almost unresponsive. I’ve tried to redeploy without success. I’ve upgraded to a higher Price tier (B1) without success.

I’ve investigated the activity via the Azure portal

But it does not tell me much. (I’m not that experienced as DevOps Engineer :-))

Maybe you can help. You can find the source code here https://github.com/marcskovmadsen/awesome-streamlit. This include detailes on the CI/ CD and Hosting.
the docker image here https://cloud.docker.com/u/marcskovmadsen/repository/docker/marcskovmadsen/awesome-streamlit
the application here https://awesome-streamlit.azurewebsites.net/

Marc · October 9, 2019, 3:00pm

It’s actually responsive again. And much more responsive as i’ve upgraded to a higher price tier.

I don’t know what the problem was. I had a lot of open tabs in Google Chrome and was opening and closing tabs while trying out the app. I had been working with the https://awesome-streamlit.azurewebsites.net/ site a lot. So my hypothesis is that somehow some connections are not closed or something.

Does that make sense at all?

Marc · October 10, 2019, 4:34am

Now the problem is there again. It seems that app gets unresponsive over time.

I will investigate but if you have any ideas they are much appreciated.

For now I will try running the Docker container locally for a longer period of time to see if I can replicate locally.

You can also run it via

docker run -it -p 80:80 --entrypoint "streamlit" marcskovmadsen/awesome-streamlit:latest run app.py

thiago · October 10, 2019, 6:55am

Hi @Marc

A few things that are worth investigating are:

Is there a memory leak? The simplest way to verify this hypothesis would be to look at a chart of memory usage vs time.
Maybe the app is getting slow because too many users are using it at the same time. Some of the sub-apps that your awesome-streamlit app executes can be very CPU intensive. So it’s very likely one user’s session can starve another’s of CPU cycles (once enough users are connected).
One way to verify this would be to look at 3 charts: average response time, # open websocket connections (or at least # of HTTP requests per time), and CPU usage. If the three charts spike at the same locations, this hypothesis looks strong.

If the problem is (1), the solution would be to debug and find the leak.

If it’s (2), a possible solution would be to replicate your app N times and put the replicas behind a load balancer. I haven’t used Azure WebApp containers before, but the marketing-speak on its official page makes it seem like they all have load balancers, and they should scale automatically. So I assume you just need to choose the min and max number of replicas you want, and Azure will do the rest.

By the way, this is one of the many reasons we think Streamlit For Teams will be useful: Machine learning apps are very CPU/GPU-intensive — especially when compared to normal web apps. And setting up the infrastructure to serve such demanding Python apps to the world is definitely not easy

Marc · October 10, 2019, 1:14pm

Thanks @thiago

I was also wondering whether there are some settings in the config file that I have not set correctly. My config file is here

github.com

MarcSkovMadsen/awesome-streamlit/blob/master/.streamlit/config.prod.toml

# Below are all the sections and options you can have in ~/.streamlit/config.toml.

[global]

# By default, Streamlit checks if the Python watchdog module is available and, if not, prints a warning asking for you to install it. The watchdog module is not required, but highly recommended. It improves Streamlit's ability to detect changes to files in your filesystem.
# If you'd like to turn off this warning, set this to True.
# Default: false
disableWatchdogWarning = false

# Configure the ability to share apps to the cloud.
# Should be set to one of these values: - "off" : turn off sharing. - "s3" : share to S3, based on the settings under the [s3] section of this config file.
# Default: "off"
sharingMode = "off"

# If True, will show a warning when you run a Streamlit-enabled script via "python my_script.py".
# Default: true
showWarningOnDirectExecution = true

# Level of logging: 'error', 'warning', 'info', or 'debug'.
# Default: 'info'

This file has been truncated. show original

If you could take a look it would be much appreciated.

I’m actually in doubt what disables livereload as I guess should be disabled in Production. I don’t think that is well described in the file.

Thanks.

Marc · October 10, 2019, 1:25pm

Memore usage is fine

Marc · October 10, 2019, 1:26pm

CPU Usage is fine

Marc · October 10, 2019, 1:27pm

Average response time is NOT OK

Marc · October 10, 2019, 1:29pm

Web socket connections I cannot find

Marc · October 10, 2019, 4:30pm

I have now find some logs from the Docker containers and I can see that exceptions like the below are raised by Tornado.

Could that be the cause?

For example

2019-10-10T07:58:12.985172340Z
2019-10-10T07:58:12.985227741Z You can now view your Streamlit app in your browser.
2019-10-10T07:58:12.985240841Z
2019-10-10T07:58:12.985250742Z URL: http://0.0.0.0:80
2019-10-10T07:58:12.985260542Z

2019-10-10T13:08:58.001240095Z Future exception was never retrieved
2019-10-10T13:08:58.001303796Z future:
2019-10-10T13:08:58.001320297Z Traceback (most recent call last):
2019-10-10T13:08:58.001330897Z File “/usr/local/lib/python3.7/site-packages/tornado/websocket.py”, line 874, in wrapper
2019-10-10T13:08:58.001345797Z yield fut
2019-10-10T13:08:58.001355898Z File “/usr/local/lib/python3.7/site-packages/tornado/gen.py”, line 1133, in run
2019-10-10T13:08:58.001365798Z value = future.result()
2019-10-10T13:08:58.001375498Z tornado.iostream.StreamClosedError: Stream is closed
2019-10-10T13:08:58.001385399Z
2019-10-10T13:08:58.001395099Z During handling of the above exception, another exception occurred:
2019-10-10T13:08:58.001404899Z
2019-10-10T13:08:58.001414299Z Traceback (most recent call last):
2019-10-10T13:08:58.001424100Z File “/usr/local/lib/python3.7/site-packages/tornado/gen.py”, line 1141, in run
2019-10-10T13:08:58.001434400Z yielded = self.gen.throw(*exc_info)
2019-10-10T13:08:58.001443900Z File “/usr/local/lib/python3.7/site-packages/tornado/websocket.py”, line 876, in wrapper
2019-10-10T13:08:58.001453800Z raise WebSocketClosedError()
2019-10-10T13:08:58.001463401Z tornado.websocket.WebSocketClosedError

Marc · October 10, 2019, 6:07pm

I also found this process list

It’s seems the CPU is running at 100%. I just don’t get that because there are not really any trafic to the container and the small things I is just showing some markdown, a table, a chart and then there is an NLP app that takes 1 second to run on my local pc.

But every proces item from 15:30 UTC to 17:30 UTC looks like that with 100% CPU.

Marc · October 10, 2019, 6:25pm

I also found this chart with CPU Time. I don’t know how to relate to it though.

and the CPU usage

Marc · October 10, 2019, 6:31pm

But again something i going in wrong according to the logs

2019-10-10T15:47:58.335547527Z Future exception was never retrieved
2019-10-10T15:47:58.335609329Z future:
2019-10-10T15:47:58.335626229Z Traceback (most recent call last):
2019-10-10T15:47:58.335636929Z File “/usr/local/lib/python3.7/site-packages/tornado/websocket.py”, line 874, in wrapper
2019-10-10T15:47:58.335647430Z yield fut
2019-10-10T15:47:58.335657430Z File “/usr/local/lib/python3.7/site-packages/tornado/gen.py”, line 1133, in run
2019-10-10T15:47:58.335667730Z value = future.result()
2019-10-10T15:47:58.335677630Z tornado.iostream.StreamClosedError: Stream is closed
2019-10-10T15:47:58.335687631Z
2019-10-10T15:47:58.335697131Z During handling of the above exception, another exception occurred:
2019-10-10T15:47:58.335706931Z
2019-10-10T15:47:58.335716431Z Traceback (most recent call last):
2019-10-10T15:47:58.335725932Z File “/usr/local/lib/python3.7/site-packages/tornado/gen.py”, line 1141, in run
2019-10-10T15:47:58.335735932Z yielded = self.gen.throw(*exc_info)
2019-10-10T15:47:58.335745632Z File “/usr/local/lib/python3.7/site-packages/tornado/websocket.py”, line 876, in wrapper
2019-10-10T15:47:58.335755832Z raise WebSocketClosedError()
2019-10-10T15:47:58.335765733Z tornado.websocket.WebSocketClosedError

2019-10-10T15:48:37.278627607Z Future exception was never retrieved
2019-10-10T15:48:37.278677208Z future:
2019-10-10T15:48:37.278701209Z Traceback (most recent call last):
2019-10-10T15:48:37.278712109Z File “/usr/local/lib/python3.7/site-packages/tornado/websocket.py”, line 874, in wrapper
2019-10-10T15:48:37.278722209Z yield fut
2019-10-10T15:48:37.278731810Z File “/usr/local/lib/python3.7/site-packages/tornado/gen.py”, line 1133, in run
2019-10-10T15:48:37.278741810Z value = future.result()
2019-10-10T15:48:37.278751310Z tornado.iostream.StreamClosedError: Stream is closed
2019-10-10T15:48:37.278760710Z
2019-10-10T15:48:37.278769811Z During handling of the above exception, another exception occurred:
2019-10-10T15:48:37.278779111Z
2019-10-10T15:48:37.278788111Z Traceback (most recent call last):
2019-10-10T15:48:37.278797111Z File “/usr/local/lib/python3.7/site-packages/tornado/gen.py”, line 1141, in run
2019-10-10T15:48:37.278806612Z yielded = self.gen.throw(*exc_info)
2019-10-10T15:48:37.278815712Z File “/usr/local/lib/python3.7/site-packages/tornado/websocket.py”, line 876, in wrapper
2019-10-10T15:48:37.278827412Z raise WebSocketClosedError()
2019-10-10T15:48:37.278836612Z tornado.websocket.WebSocketClosedError

2019-10-10T15:49:56.350452137Z Future exception was never retrieved
2019-10-10T15:49:56.350500139Z future:
2019-10-10T15:49:56.350513939Z Traceback (most recent call last):
2019-10-10T15:49:56.350524239Z File “/usr/local/lib/python3.7/site-packages/tornado/websocket.py”, line 874, in wrapper
2019-10-10T15:49:56.350534539Z yield fut
2019-10-10T15:49:56.350544040Z File “/usr/local/lib/python3.7/site-packages/tornado/gen.py”, line 1133, in run
2019-10-10T15:49:56.350554240Z value = future.result()
2019-10-10T15:49:56.350563740Z tornado.iostream.StreamClosedError: Stream is closed
2019-10-10T15:49:56.350573540Z
2019-10-10T15:49:56.350583041Z During handling of the above exception, another exception occurred:
2019-10-10T15:49:56.350592741Z
2019-10-10T15:49:56.350601941Z Traceback (most recent call last):
2019-10-10T15:49:56.350611341Z File “/usr/local/lib/python3.7/site-packages/tornado/gen.py”, line 1141, in run
2019-10-10T15:49:56.350621042Z yielded = self.gen.throw(*exc_info)
2019-10-10T15:49:56.350630542Z File “/usr/local/lib/python3.7/site-packages/tornado/websocket.py”, line 876, in wrapper
2019-10-10T15:49:56.350640242Z raise WebSocketClosedError()
2019-10-10T15:49:56.350649642Z tornado.websocket.WebSocketClosedError

2019-10-10T16:06:03.641717003Z Future exception was never retrieved
2019-10-10T16:06:03.641765704Z future:
2019-10-10T16:06:03.641779005Z Traceback (most recent call last):
2019-10-10T16:06:03.641805405Z File “/usr/local/lib/python3.7/site-packages/tornado/websocket.py”, line 874, in wrapper
2019-10-10T16:06:03.641816706Z yield fut
2019-10-10T16:06:03.641826106Z File “/usr/local/lib/python3.7/site-packages/tornado/gen.py”, line 1133, in run
2019-10-10T16:06:03.641835906Z value = future.result()
2019-10-10T16:06:03.641845306Z tornado.iostream.StreamClosedError: Stream is closed
2019-10-10T16:06:03.641854707Z
2019-10-10T16:06:03.641863907Z During handling of the above exception, another exception occurred:
2019-10-10T16:06:03.641873307Z
2019-10-10T16:06:03.641882307Z Traceback (most recent call last):
2019-10-10T16:06:03.641891608Z File “/usr/local/lib/python3.7/site-packages/tornado/gen.py”, line 1141, in run
2019-10-10T16:06:03.641901008Z yielded = self.gen.throw(*exc_info)
2019-10-10T16:06:03.641910008Z File “/usr/local/lib/python3.7/site-packages/tornado/websocket.py”, line 876, in wrapper
2019-10-10T16:06:03.641919408Z raise WebSocketClosedError()
2019-10-10T16:06:03.642059212Z tornado.websocket.WebSocketClosedError

Marc · October 10, 2019, 6:44pm

I’ve run the docker container locally for 30 minutes and with one connection open in Chrome. When I start the app in docker it takes a fraction of a second to navigate from Home to the Vision Page.

After the Docker container has run for 30 minute it takes more than 8 seconds to navigate from Home to the Vision Page

Looking at the Task Manager I don’t see any CPU usage explosion. Its around 20% (of my laptops CPU) all the time.

thiago · October 10, 2019, 10:31pm

Ooooh that’s really interesting.

I created a bug so someone here can investigate. Can you doubple-check that repro steps listed in the bug are correct?

Marc · October 11, 2019, 4:08am

FYI @thiago. I’ve added my comments to the repro steps.

I’ve created an issue on the awesome-streamlit repo

Marc · October 18, 2019, 6:51am

FYI @thiago

Problems have now been solved and the performance is awesome!

Check it out at https://awesome-streamlit.org/

thiago · October 18, 2019, 10:30pm

That’s amazing!

Can you share what you did to fix it? Was there an issue with the app’s code, or a Streamlit issue, or something in between?

Marc · October 19, 2019, 1:35am

I did a lot of changes. But I believe the essential parts are

I turned Always On on in Azure
I created a script that runs inside my docker container to ping awesome-streamlit.org every 5 minutes
I removed the custom import_lib.reload functionality I was using (for development) to reload deeply nested modules. See the issue here https://github.com/streamlit/streamlit/issues/366
I set folderWatchBlacklist = ['']in the config.toml file.

In order to investigate the issue I had to add detailed logging to my docker continer, especially timestamp every line into the log and log top resource usage every 1 minutes. I have a request for time in the loggingFormatter here https://github.com/streamlit/streamlit/issues/447.

I believe item 1 and either 3 or 4 did the trick. But I have not reproduced. Just happy that it works after so many investigations.

I’ve recorded the full investigation here

Topic		Replies	Views
Streamlit on Azure Deployment azure , configuration	10	7095	August 3, 2021
Streamlit-App on Azure Web App for Container (with Docker) gives just a "Please Wait..." message Deployment azure , docker , debugging	5	1075	June 5, 2024
How to deploy streamlit on Azure? Deployment azure , configuration	17	21803	October 19, 2021
Streamlit on Azure: Webapp stuck at "Please wait ..." Deployment azure	7	2256	October 3, 2023
Failed: Deployment on Azure WebAPP via Docker Container Deployment windows , azure , docker	2	1865	May 13, 2022

Streamlit App deployed as Azure WebApp for containers becomes unresponsive over time

Related Topics

Hello there 👋🏻

Cookie settings

Strictly necessary cookies

Performance cookies

Functional cookies

Targeting cookies