App resets every four minutes in Gcp cloud run

Using Streamlit
1

I have deployed my streamlit app on Google cloud run (Dockerized and everything).

The app is essentially doing some llms calls and that takes a while to finish (10+ mins), so there is a progress bar while it’s running. Locally I have no problem running it for even hours in a row, but in cloud run there are some get /health requests (and other endpoints) that happen roughly every 240 seconds (that is the maximum that can be set) and when that happens the streamlit app basically resets exactly as I had just refreshed the page.

here are some logs of the requests:

2024-10-08 10:40:30.609 CEST
GET304242 B2 msChrome 123 https://yunoai-mergeracquisition-streamlit-demo-*****.europe-west1.run.app/_stcore/health

2024-10-08 10:40:30.857 CEST
GET101799 B301 sChrome 123 https://yunoai-mergeracquisition-streamlit-demo-*****.europe-west1.run.app/_stcore/stream

2024-10-08 10:45:31.968 CEST
GET304154 B2 msChrome 123 https://yunoai-mergeracquisition-streamlit-demo-*****.europe-west1.run.app/_stcore/host-config

I think these automatic health/uptime checks are the culprits which are resetting my application, and unfortuntely there is nothing I can do on the cloud run side as these cannot be deactivated …

Does anybody know how to solve this? Perhaps something can be done on the application side to prevent the checks to interrupt everything?

thank you!

2

Hey @Camillo ,

I am not sure about the issue you’re having, but it might be worth looking into the Request timeout of the Cloud Run instance.
If you go to the cloud run instance, and press:

  • Edit & Deploy new version

You can adjust the Request timeout setting. The maximum time that can be set is 3600 seconds, which should be enough for your needs.

Try it out, and see if this solves your issue!

Good luck!

3

Thanks Dirk

I actually already set that to 3600 secords, cause I thought that might be one of the issues, but it did not solve the problem :frowning:

4

Hi @Camillo ,

Too bad that this wasn’t the solution! After some more reading, I now also see what you mean with the maximum time of 240 seconds for the health check. It seems like the app is just too busy during the time the health check is happening.

One other thing I saw in the settings is that you can also set a failure threshold. You might be able to set this to n (higher than the default of 1), so that you have x*240 seconds for the process to finish. I’m not entirely sure if this works, but it might be worth a try.

What you could try as well is to run your llms calls as a seperate process, while keeping the streamlit process available.

Hopefully one of these options help!

5

Ciao Dirk, sorry for not updating the thread but that was indeed the solution, set the failure threshold to 10 and now it takes 2400 secs for the app to reset, which is more than enough for me!

best,

Camillo

1 Like
6

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.