@Shawn_Pereira, what helped me to build large apps is this benchmarking approach here: Benchmarking a streamlit app
I leave this code permanently in my app and wrap it with a feature toggle if.
In this way you can quickly run small benchmarks by just setting a bool to true and get a pretty good feeling about where your code is slow.
If you focus on the right parts of your code, you can even run large amounts of code quite quickly.
I also store my data in optimized dataframes stored in parquet files that can be read very quickly. This can even be quicker than caching the dataframe.
Another point is that you need caching but you should not overdo it and not use it for every function, especially for large dataframes.
For me it has worked out to use kind of a function chaining where each function calls the previous one and also gets some settings from userinput, usually as a dict. In this way you can use caching for some of them and only rerun the calculations that need to be rerun based on the new user input.
I also made some performance improvements on the aggrid component to better suit my application. Things like this can also be necessary if you find your bottleneck not to be in your own code.