Hi! I’ve share my first streamlit app, which generates music lyrics with AiTextGen. After a couple of hours I always have to reboot it because it says it has gone over the limits. I don’t really understand what’s making it heavy, as the app only generates text and writes it on the screen, its not saving variables.
If you can wrap this line in st.cache, it should prevent the scenario where multiple simultaneous users load the model, which could be quite costly in terms of resources (each user gets a separate session in the container).
Thanks for taking the time to answer. But which line are you referring too? I was thinking that maybe the loading the model was the problem, but I’m not sure if I should put the st.cache at importing packages and the model, or in the generation process.
I’m assuming Randy meant wrapping st.cache when you’re loading the model. It will prevent the script from re-loading the model with each widget interaction. Additionally, it will reuse the same cached model across multiple simultaneous users, rather than loading a separate 30 MB model into RAM for each user.
Here’s an example:
ai = aitextgen(model_folder="trained_model", tokenizer_file="aitextgen.tokenizer.json")
ai = load_model()
Implementing the above will also result in an error related to st.set_page_config. The page config command should be the first Streamlit command that is executed. As such, you should move it under your package imports.
To make it easier for you, I’ve cloned your repo and made the required changes here