How do you create data pipelines from streamlit to cloud services running your code separately?

How can I create a data pipeline from a streamlit app to a cloud service that makes a 9 hours scraping computation and if the cloud service is finished send it back to my streamlit app.

The dilemma is my code works as a streamlit app but stops scraping after 15 mins because any streamlit app will stop its computation after 15-20 mins runtime because that’s how web sockets work.

I have no idea how to create those data pipelines or even how make streamlit ask the cloud service if it’s finished

What cloud service can you recommend? Python Anywhere, Firebase, google cloud…?

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.