Hack: Check IF its cache-miss before calling that expensive function

[smartm13]

Hi guys,
I have put together following code piece to provide the ability to pre-emptively check if any cached function has already cached a particular set of arguments or not.

Use-case: I have a model config as input and predictions as output. I let users set the config and then they press button to see output. And it takes time to run the model.
Sometimes, users choose a config which was already trained upon. But they have no way to know that without actually pressing the button and having results instantaneously since results for that config were cached and persisted. But pressing that button all the time could risk actually running the whole model again, if it was a cache miss (a config appearing first time).

I researched a bit, but found no-way around. So, came up with this dirty trick:

Define this 2 decorators first: st_cache_monitor and st_cache_monitored.
Later declare your cached function with that decorator. For example:

@st_cache_monitored
@st.experimental_memo(...)
@st_cache_monitor
def expensive_func(a, b=None, **extras):
    # returning after heavy time.sleeps

And now you can check if its expensive to call the func without actually calling it.
The decorated function gets an additional callable attribute query_cache which accepts the same arguments as the original function and returns hit if given args are already in cache, else miss.

Example Usage:

if expensive_func.query_cache(4.4, [3,2,1] , kwarg_c=4) == "miss":
    st.text("Your config is not in cache. Press button to have results")

if expensive_func.query_cache(4.4) == "hit":
    st.text("No need to wait, your results are already available.")
    st.write(expensive_func(4.4))

I am not sure if anybody else needs this facility, but I did. The caching module seemed well-abstracted by streamlit, and I liked it that way. Hence, this.

Notes:

• This st_cache_monitor works seamlessly only with st.experimental_memo (st.cache).
• It carefully leverages the fact that Exceptions thrown by a cached function don’t result in cache storage and arguments that start with _ are not changing the hashed key.

Update: Realized that the earlier version wasn’t working that well, so did split that into 2 decorators.

[randyzwitch]

This is really fascinating @smartm13, thanks for sharing!

[willhuang]

Agreed! That is so cool @smartm13 !!! Thanks for posting!