Asked again since no one answered (Http 429 error only when hosted it is fine locally)

[sully20832]

Right now the app is fine but once a lot of people start using it I start getting http 429 errors since I scrape results from google as part of my search method. How do I ensure that this does not happen again ??

I asked chatgpt and it said:

from streamlit.script_runner import StopException, Throttler is it supported in the newest streamlit version, if not what are the alternatives. I would like to use it as my web app involves http resquests to google and I would like to rate limit in my Streamlit app to ensure that only a limited number of requests are sent at a time. Because I get 429 http responses when hosting and lots of people are using my web app and locally I never face this error.? How would I avoid 429 in streamlit deploying when doing http requests

My app: https://adamaivideosearch.streamlit.app/

[mathcatsand]

I’m not sure how much anyone can say without seeing your code.

HTTP 429 is literally “too many requests” so I would think one or more of the urls you are scraping simply won’t allow high volume traffic. If you are on a free service like Streamlit Cloud, I don’t think you have a guarantee that traffic from your app will be viewed distinctly from traffic from other apps, which can complicate things.

Any website or API in existence may enforce whatever limits on use they desire. If you are certain your app by itself is not violating those limits, then I think you’d need to host some other way to ensure traffic from your app is not mixed in with other traffic that may be coming from the same servers.