How to sync Streamlit with background tasks: Outer/Inner caching

Using Streamlit
,
1

Hi everyone,

I am building an app on a virtual machine I have structured my architecture to run as two completely separated processes in two different terminals:

  1. Terminal 1 (Background Runner): Periodically processes raw data using heavy LLM translation/extraction tasks, outputting a cleaned alerts.jsonl file.

  2. Terminal 2 (Streamlit UI): A presentation layer dashboard running 24/7 that reads that generated file and displays charts/maps.

The Goal:

Since Streamlit is running continuously, I want it to load the pre-processed data instantly from memory for my users. However, the exact millisecond my background runner completes a new run and updates the alerts.jsonl file, I need Streamlit to instantly detect this change and sync without requiring users to wait out a long static TTL (like 24h).

My Proposed Strategy (The Outer/Inner Function Pattern):

I want to use a non-cached outer function to check the file state, which then passes the modification timestamp as a “cache invalidation key” to a cached inner function:

Python

import os
import pandas as pd
import streamlit as st

# 1. Outer function (Runs on every user interaction/refresh)
def load_and_process_data():
    alerts_path = "results/alerts.jsonl"
    
    # Get the file's last modified timestamp as our cache key
    mtime = os.path.getmtime(alerts_path) if os.path.exists(alerts_path) else 0.0
    
    # Pass the timestamp to the cached worker function
    return _cached_load_and_process(mtime)

# 2. Inner function (Cached)
# We set max_entries=1 so that the moment a new timestamp forces a reload,
# the old dataset is immediately evicted from the server's RAM.
@st.cache_data(ttl=86400, max_entries=1)
def _cached_load_and_process(timestamp):
    alerts_path = "results/alerts.jsonl"
    centroids_path = "src/dashboard/country_centroids.csv"
    
    with open(alerts_path, "r") as f:
        df_alerts = pd.read_json(f, lines=True)
    
    with open(centroids_path, "r") as f:
        df_centroids = pd.read_csv(f)

    df_alerts["pub_date"] = pd.to_datetime(df_alerts["pub_date"]).dt.date
    
    df_alerts_mapped = pd.merge(
        df_alerts, 
        df_centroids, 
        left_on="iso2_code", 
        right_on="iso_alpha2", 
        how="inner"
    )
    return df_alerts_mapped

  1. Do I actually need ttl=86400 if I am already using max_entries=1? Since the cache size is capped at 1, does setting a TTL add any benefit (like clearing memory when the app is idle), or is it redundant because the timestamp parameter handles invalidation automatically anyway?

  2. Is this “Outer/Inner” pattern still the standard/recommended approach for syncing Streamlit with external background tasks modifying local files?

  3. Is using max_entries=1 the safest way here to prevent RAM bloating when the background worker generates new files over time? Or is there a cleaner built-in method I’m missing?

  4. Eventually, I will migrate my database to MongoDB. Does this pattern transition smoothly if I replace os.path.getmtime() with a function that fetches the newest document’s _id from Mongo?

Would love to hear how other developers handle instant cache invalidation on file/DB updates. Thank you!

2

Welcome to the Streamlit community, and thanks for the detailed question! :balloon: Your “outer/inner” function pattern—using a non-cached outer function to pass a file modification timestamp as a cache key to a cached inner function—is a well-established and recommended approach for syncing Streamlit with external file changes. This ensures the cache is invalidated and data is reloaded instantly when the file updates, without relying on a static TTL. Setting max_entries=1 is a good safeguard to prevent RAM bloat, as only the latest dataset is kept in memory, and old entries are evicted as soon as a new timestamp is detected. The ttl parameter is not strictly necessary here, since cache invalidation is handled by the timestamp argument; ttl would only be useful if you want to force a refresh after a certain period regardless of file changes, or to clear memory if the app is idle for a long time, but with max_entries=1 and your pattern, it’s mostly redundant.

This pattern transitions smoothly to a database like MongoDB—just replace the file mtime with a unique, always-increasing value (such as the latest document’s _id or an updated_at timestamp) as the cache key. Streamlit does not currently provide a built-in file watcher or database trigger for cache invalidation, so your approach is the standard way to achieve instant cache refresh on external updates. For more, see the discussion and best practices in the Streamlit caching docs and community forum.

Sources:

3

Hi! unfortunately it does not work as expected … the only way I could have it updated is to set a refresh .. for now it is set every 15 minutes

# Refresh every 15 minutes * 60 seconds * 1000 milliseconds = 900000
st_autorefresh(interval=15 * 60 * 1000, key="quarter_hour_heartbeat")

Maybe it is not the ideal solution but it is a sort of workaround. … the data are updated every 24 hours but I fear using 24 hours as refresh because it may not pick the exact time when they are updated.

Should I try to use the previous approach but without the ttl ? could the ttl prevent the new data coming up ?
It is important to note that load_data is defined in another python script and I am importing this function in the main script called app.py

Thank you for your help
Best wishes
Angela