I have a forecasting application which needs data updated every 24 hours. I set up a CI/CD pipeline to run at every 24 hours and to create the needed csv files.
I now need to run some models on the created csv files, as well at 24 hours. I was thinking to use st.cache for the run of models and creation of pickle files for each model, but I’m afraid this is not the best optimal solution. Is there any way to set up a specific time for the st cache to run, like overnight or something so that the users are not affected by this creation of pickle files? -Because it might take some minutes for the models to run and the pickle files to be created.
Also, should I also add the script which creates the csv files in a st.cache method, so to make sure it runs and is available for the models to run?
How else could I solve this issue, so that my pickle files are fresh every day?