I have a streamlit app (built in docker, deployed to heroku) that formulates an optimization (a linear program) and then sends the schedule to a gurobi server. However, the process takes a while, and thus for a 20-year optimization at the hourly level, I chunk it into years (8760 data points for a year, x20 years).
Basically, once the inputs are set by the user and saved as state variables a la Joel Grusā Game State hacks , the process is:
Formulate inputs and constraints to optimization for 2021 ā
send to gurobi ā
get back optimal solution from gurobi as a dataframe->
display on Streamlit app āOptimal Solution for 2021 Complete! Moving on to 2022ā¦ā
Repeat for next iteration, for 20 iterations.
Concatenante all 20 dataframes from the process and display some summary results and some download and āSend to databaseā buttons for the results.
Each iteration takes about a minute. However, I can only make it to about the 10th iteration (sometimes less, sometimes more) of that before the streamlit frontend ādisconnectsā from the running loop process of the app and resets all of my state variables, etc, like I had just reloaded the page. The background app process still continues (itās still looping through the iterations - it doesnāt know streamlit got disconnected), but there is no way to reconnect to the running process again. Without streamlit, this process is fine to run locally with scripts, but with streamlit the frontend is so unreliable as to be unusable for this app. Nothing more frustrating than to get nearly done with a process that took 20 minutes and then it resets itself, forcing the user to start the process over.
Iām really not sure what to try at this point, other than build something much more robust like a flask app that sends these jobs to a job container, which then sends back the result when done. Running the optimization on the same computer as the frontend seems like a foolās errand.
Anyone have tips to keep a streamlit frontend connected to a long-running process without resetting?
In my experience those symptoms sound a lot like thereās some unexpected HTTP timeout going on on the host side. The only thing that gives me pause here is that the default HTTP timeout is ~30s, so usually thatās how long your app would take between disconnect-reconnect cycles. But in your case it sounds like these reconnect-disconnect cycles are much longerā¦
Either way, to help debug it would be great to get a couple of things:
If you look at your server logs, do you see any errors?
On the browser-side, if you open your browserās dev tools, do you see anything either on the JS console or in the Network tab?
If you donāt know what to look for, or just want a second pair of eyes, feel free to post the logs from (1) and a HAR file from (2) here so I can help debug.
Thanks for the response, sorry about the late reply. Iāve recreated the error and made a screen recording, and I did so running the app locally to avoid the confounding variable of the heroku-based deployment, which I played up too much in the initial post. It actually is more likely to fail running locally, though Iām not sure why. For a 20 year run (~10 minutes), it resets the app about 60% of the time somewhere along in the process (I had to record a couple times to show a failure, as the first couple examples made it through the analysis successfully). Link to recording
I have a HAR file for a later failure (not the one in the video) but the forum wonāt let me attach it. Should I convert it to some other file type? We donāt need the Heroku logs because since I ran it locally. And now that I think about it, it could be a memory thing - since python is pretty bad at memory management and Chrome is a RAM hog, perhaps running it several times successfully caused it to fail? Though I CTRL+C stopped the streamlit app and started it again each time and my memory usage on my computer seemed pretty static. Iām at a loss. ::
After watching the video I tried reproducing the issue locally using a toy example, but Iām just not able to
This is the toy example I tried:
import streamlit as st
import time
import datetime
"""
# Long-running app example
"""
start_time = datetime.datetime.now()
for i in range(20):
"---"
"Loop number:", i
"Ellapsed time:", datetime.datetime.now() - start_time
# Tried with
# time.sleep(10)
# ...but can't repro
# Tried with
# time.sleep(60)
# ...but can't repro
At least this rules out a few hypotheses I had about websocket keepalive failuresā¦
Thanks for the help. Iāve uploaded the .HAR file and the bug.log from the most recent run to that folder (at least I believe itās that folder - let me know if you canāt see them). I ran it again locally in a chrome profile with no extensions and it failed at about the same point as in the video.
I believe the failure happens around line 2977 of the log. You can see ~ line 2958 āoptimization is solved with status 2ā which is good, that means that year completed (Year 1 of 20), and is saving the results as a dataframe (see the pandas warning about setting a value on a slice from a df blah blah etc). Then on line 2977 we get āServer state: State.ONE_OR_MORE_BROWSERS_CONNECTED ā State.NO_BROWSERS_CONNECTEDā which I believe is our issue.
I will attempt to run it to completion and post what a successful log file looks like as well, as there is a lot going on in that log.
(Note, neither here nor there but I also upgraded to streamlit 0.8.0 before that run and the log files attest to me being excited about using secrets! )
I believe the failure happens around line 2977 of the log.
Yup, thatās right.
The logs around that line make me think that the problem is on the browser side. We basically see the websocket disconnecting the same way it would if the browser had terminated the connection normally. Very odd!
Unfortunately, the data in the HAR file starts a few seconds after the websocket first connects, which means the websocket connection (and, more importantly, its disconnection!) isnāt present in the data
Can you try capturing a HAR file using the method below?
Start your Streamlit server streamlit run foo.py
Open a browser at localhost:8501
Open the dev tools in the browser > Network > āpreserve logā
Reload the browser page with Ctrl-R (or Cmd-R)
Now save the HAR file and upload to our shared Drive link
Also: can you copy/paste into a file whatever you see in the Console tab in the dev tools?
If none of this bears fruit, our next best step is to set up a video call and see whatās up!
Okay, @thiago Iāve hopefully captured everything in a couple newly uploaded .HAR files from similarly failed runs.
There is one from just recording after the app has started, and one from a full refresh, and both gave similar errors when the app reset itself.
In failed_run_attempt2, there is a 101 code on ~line 71, that followed 304s at line 61 and 70, that all seemed to be generated just as the app reset itself.
The file failed_run_from_full_refresh is the similar, but started a bit āfurther backā from a Cmd-R Refresh point. We see the 304 and 101 code combo around line ~131. Hopefully that helps disentangle what is normal process vs what is failure.
And here is the full dev tools console capture for failed_run_attempt2 (also uploaded to the shared drive). Thanks for working through this with me!
Hello @thiago !
Iām having exactly the same issue as @Kladar. I have a model that takes almost an hour to run and sometimes it just disconnects from the backend process while I see the console is still running but the front is just reset to default. Is there a more friendly solution already for this?
Hey @Kladar and @avila196 - Iām taking a look at this now. Itās obviously a trivial change; the question is, will it have other consequences that weāre not considering?
Iāll do a bit more research - but the plan right now is to merge this change in the next 2 weeks.
It could be a memory leak. We havenāt had the problem since the websocket ping patch in 0.84, but we also havenāt upgraded to 1.8 yet. Saw this recently, maybe it will help: 3 steps to fix app memory leaks
Thanks for stopping by! We use cookies to help us understand how you interact with our website.
By clicking āAccept allā, you consent to our use of cookies. For more information, please see our privacy policy.
Cookie settings
Strictly necessary cookies
These cookies are necessary for the website to function and cannot be switched off. They are usually only set in response to actions made by you which amount to a request for services, such as setting your privacy preferences, logging in or filling in forms.
Performance cookies
These cookies allow us to count visits and traffic sources so we can measure and improve the performance of our site. They help us understand how visitors move around the site and which pages are most frequently visited.
Functional cookies
These cookies are used to record your choices and settings, maintain your preferences over time and recognize you when you return to our website. These cookies help us to personalize our content for you and remember your preferences.
Targeting cookies
These cookies may be deployed to our site by our advertising partners to build a profile of your interest and provide you with content that is relevant to you, including showing you relevant ads on other websites.