How best to apply a save on_change function to a dynamic filter?

Using Streamlit
, ,
1

I’m trying to make an app that loads a df that will later be used as input data further down the workflow. It’s quite large so the user will need to be able to apply filters and check boxes to mark what rows they want carried forward.

My problem is trying to preserve the user’s choices between changing the filters. I’ve tried implementing session states and following guides (python - Retaining changes for Streamlit data editor when hiding or switching between widgets/pages - Stack Overflow) but I’ve not been able to get it to work. When the edited df saves over the original df it also applies the filter, effectively deleting everything filtered out. Also, as I’m using dynamic_filters instead of changing page (as in the guide), I’m struggling to find a good place to add an on_change. I fell into the pitfall of having the on_change in the data_editor so it has the “have to apply all changes twice” flaw.

I’m working on fixing the filter overwrite by only updating the changed values, but I’m still new to python so I’ve not been having much luck with this so far, in part because it has been hard to test with the current on_change being in the data_editor. As for the on_change issue, ideally, I’d prefer not to make a separate button that saves the changes before applying the filters as this is going to impact the UX a lot. If anyone knows a good way to apply an on_change to dynamic_filters, I’d really appreciate it.

I’ve been testing in the streamlit playground while I get to grips with it, here is the code so far.

import streamlit as st
import pandas as pd
import numpy as np
from streamlit_dynamic_filters import DynamicFilters

num_rows = 500
np.random.seed(42)
data = []

if 'df1' not in st.session_state :
    for i in range(num_rows):
        data.append(
            {
                "Preview": f"https://picsum.photos/400/200?lock={i}",
                "Views": np.random.randint(0, 1000),
                "Active": np.random.choice([True, False]),
                "Category": np.random.choice(["🤖 LLM", "📊 Data", "⚙️ Tool"]),
                "Progress": np.random.randint(1, 100),
            }
        )     
    st.session_state.df1 = pd.DataFrame(data)
    st.session_state.edited_df1 = st.session_state.df1.copy()
    
# Convenient shorthand notation
df1 = st.session_state.df1

# Save edits by copying edited dataframes to "original" slots in session state
def save_edits():
    merged_df = pd.merge(st.session_state.df1,st.session_state.edited_df1,how='outer',indicator=True)
    #st.session_state.df1 = st.session_state.edited_df1.copy()
    diff_df1 = merged_df[merged_df['_merge'] == 'right_only']
    st.write(diff_df1)
    #[INSERT CODE THAT FINDS EDITED VALUES AND UPDATES relevant rows of df1]        
   
if st.toggle("Show df", on_change=save_edits):
        
   dynamic_filters = DynamicFilters(df1, filters = ['Category','Active']) #on_change here raises errors
   with st.sidebar :
       st.write("Dataframe Filters")
       dynamic_filters.display_filters() #on_change here raises errors
       progress = st.slider("Progress", 1,100, 100,on_change=save_edits)
        
    filtered_df = dynamic_filters.filter_df()
           
    
    st.session_state.edited_df1 = st.data_editor(filtered_df[filtered_df['Progress']<progress], on_change=save_edits, hide_index=True, use_container_width=True, num_rows="dynamic") #Currently includes the on_change but this needs to be moved elsewhere
2

Welcome to the Streamlit community, and thanks for sharing your code and a clear description of your challenge! :balloon: You’re right: preserving edits and selections across dynamic filters in Streamlit is tricky, especially with large data and when using st.data_editor and custom filters. The main issue is that when you filter, the edited DataFrame only contains the visible rows, so saving it directly will overwrite or “delete” filtered-out rows. Also, putting on_change on the data editor leads to the “double edit” problem due to Streamlit’s rerun model.

Best Practice:
To persist edits and selections across filters, you should:

  1. Store the full, unfiltered DataFrame in st.session_state.
  2. When the user edits the filtered view, update only the corresponding rows in the full DataFrame.
  3. Avoid using on_change on the data editor; instead, use a callback or button to commit changes, or update the session state immediately after editing.

Here’s a minimal, reproducible example that demonstrates this pattern:

import streamlit as st
import pandas as pd
import numpy as np
from streamlit_dynamic_filters import DynamicFilters

num_rows = 500
np.random.seed(42)
if 'df1' not in st.session_state:
    data = []
    for i in range(num_rows):
        data.append({
            "Preview": f"https://picsum.photos/400/200?lock={i}",
            "Views": np.random.randint(0, 1000),
            "Active": np.random.choice([True, False]),
            "Category": np.random.choice(["🤖 LLM", "📊 Data", "⚙️ Tool"]),
            "Progress": np.random.randint(1, 100),
        })
    st.session_state.df1 = pd.DataFrame(data)

df1 = st.session_state.df1

dynamic_filters = DynamicFilters(df1, filters=['Category', 'Active'])
with st.sidebar:
    dynamic_filters.display_filters()
    progress = st.slider("Progress", 1, 100, 100)

filtered_df = dynamic_filters.filter_df()
filtered_df = filtered_df[filtered_df['Progress'] < progress]

# Show editable table
edited_df = st.data_editor(filtered_df, hide_index=True, use_container_width=True, num_rows="dynamic", key="editor")

# Commit edits back to the full DataFrame
if st.button("Save changes"):
    for idx, row in edited_df.iterrows():
        # Find the corresponding index in the original df
        orig_idx = df1.index[df1['Preview'] == row['Preview']]
        if not orig_idx.empty:
            st.session_state.df1.loc[orig_idx, :] = row

st.write("Full DataFrame (persisted):")
st.write(st.session_state.df1.head())

Key Points:

  • Edits are only committed to the full DataFrame when the user clicks “Save changes”.
  • The original DataFrame in session state is never overwritten by a filtered subset.
  • This avoids the “double edit” bug and preserves edits across filter changes.

For more advanced UX, you could use a custom callback or experiment with Session State patterns to auto-save, but a button is the most robust and user-friendly approach for now.

Sources:

3

One small update, I did some extra research and discovered that using the below code for applying changes prevents the filters from overwriting the entire df.



def save_edits():
     merged_df = pd.merge(st.session_state.df1,st.session_state.edited_df1,how='outer',indicator=True)
     diff_df1 = merged_df[merged_df['_merge'] == 'right_only']
     if len(diff_df1)>0:
        df1.update(st.session_state.edited_df1

Streamy’s suggestions to “use a callback” or “commit changes immediately after editing” is pretty much what I’m trying to do. A “Save changes” button doesn’t help if all the changes were removed when a new filter was applied and having to click save after every change sounds like I have a grudge against users’ carpal tunnels.

As I mentioned in my original post, an “apply filter” button that includes the save edits function could be a workaround and I’ve been using that to get more accurate testing but I’d like to know if there is a more seamless way. Especially as I’ve had some issues with the filter button; if I have the original df outside of the if then I constantly show the original, if it’s as an else, anytime I click on a value that isn’t apply filter, it removes the filters. I’m going to try and fix this by having a 2nd if that will check to see if there is a filtered table but this feels very suboptimal.

`with st.sidebar :
    st.write("Facility Filters")
    dynamic_filters.display_filters()
    progress = st.slider("Progress", 1,100, 100)
    Apply_filt = st.button('Apply filters')
if Apply_filt :
    save_edits()
    filtered_df = dynamic_filters.filter_df()`
    st.session_state.edited_df1 = st.data_editor(filtered_df[filtered_df['Progress']<progress], hide_index=True,  column_config=config, use_container_width=True, num_rows="dynamic")