Updating data in in editable dataframe (st.experimental_data_editor)

[bilalmussa89]

Summary

Hi, I have an editable dataframe which I can use to select / unselect variables for my regression model and I would like to update it with the model coefficients after I run the regression. I have tried df.update but that doesnt work. Can someone help point me in the right direction? My code is below.

Steps to reproduce

Code snippet:

import streamlit as st
import numpy as np
import requests
import json
import pandas as pd
import ast
import statsmodels.api as sm
from st_aggrid import GridOptionsBuilder, AgGrid, GridUpdateMode, DataReturnMode

st.set_page_config(layout='wide')

st.write("**By: [Bilal Mussa](https://www.linkedin.com/in/bilalmussa/)**")


@st.cache_data 
def tidy_data(data):
    #clean up some of the data where needed
    data['date'] = pd.to_datetime(data['date']).dt.date
    data= data.fillna(0)
    return data

st.sidebar.write('Please load your data in')

user_input = st.sidebar.file_uploader("Upload CSV",type=['csv'])

raw_data = tidy_data(pd.read_csv(user_input,parse_dates=['date'], dayfirst=True))   

model_vars = list(raw_data.columns)
model_vars.remove('date')

st.sidebar.write('Please select your dependant variable')

option = st.sidebar.selectbox('Select your dep var?',
                      (model_vars))

st.sidebar.write('You selected:', option)

# Now we will create the model spec page/editable dataframe
model_specs = pd.DataFrame(model_vars, columns =['variable'])
model_specs['log']=0
model_specs['Adstock']=0
model_specs['DimRet']=0
model_specs['Coef.']=0
model_specs['Std.Err.']=0
model_specs['t']=0
model_specs['P>|t|']=0
model_specs['Override']=1
model_specs['Fixed']=0
model_specs['DecompGroup']=''

edited_df = st.experimental_data_editor(model_specs)
edited_df = edited_df.set_index('variable')
selected_vars = list(edited_df[edited_df['Override']==1].index)

y = raw_data[option]
x = raw_data[selected_vars]

model = sm.OLS(y, x).fit()
predictions = model.predict(x) 
LRresult = (model.summary2().tables[1])
LRresult= LRresult.rename_axis('variable')

print_model = model.summary()
st.text(print_model)
print(print_model)
edited_df.update(LRresult)

If applicable, please provide the steps we should take to reproduce the error or specified behavior.

Expected behavior:

I expect the editable dataframe to now be updated with the coefficient values. It shows on python but doesnt show on streamlit.

Actual behavior:

The coefficients, std errors etc just remain null.

Debug info

• OS version: Mac
• Browser version: Chrome

[blackary]

Could you please provide a sample csv which can be used to reproduce this issue? That would make it easier to help debug.

Without being able to see it in action, it’s a bit hard to be sure sure of exactly what issue you’re expecting to see. Are you expecting the value of edited_df to be changed in last line of your script? And, are you actually printing the value of edited_df anywhere? For example, st.write(edited_df) – I don’t see where you’re actually expecting to see the change.

[bilalmussa89]

Hi,
here is a link to the csv
https://raw.githubusercontent.com/bilalmussa/random_scripts/main/data.csv

I expect the edtiable dataframe to update with coefficients, std error, t stats etc using the LRresult dataframe.
yes, in the last line of the code i expected the edited_df to be changed/updated

[blackary]

It seems like it’s working to me. I modified the end of the script to write out the contents of edited_df before and after the .update call

st.write(edited_df)
edited_df.update(LRresult)
st.write(edited_df)

And the dataframe is getting updated

[bilalmussa89]

I see that, however, i want it to update on the 1st dataframe rather than have a second. Is that not possible?

[blackary]

edited_df is a variable which you are actively changing. You are doing the equivalent of this:

x = 6 # set the initial value
print(x) # prints 6
x = 7 # Update the value
print(x) # prints 7

If you think of your streamlit app as a script that runs from top to bottom, that might help explain why the st.write(edited_df) before the .update call is not updated. The previous values of variables do not get automatically overwritten before they were changed.

[bilalmussa89]

@blackary

st.write(edited_df)
edited_df.update(LRresult)
st.write(edited_df)

Ok. I think the other solution is to split the pane into two columns and have the dataframes side by side. one which people can edit and the other which returns the results.

[blackary]

Ah, I think I might understand better what you’re looking for. I think having two columns side-by-side is a reasonable solution.

Another way to do it is to update the original dataframe in place by using session state, but I think you’ll find this works better if you explicitly add a button to trigger the rerun.

Here’s a slightly modified script that adds a button to recalculate the values, and calls experimental_rerun to update the whole page.

import pandas as pd
import statsmodels.api as sm
import streamlit as st

st.set_page_config(layout="wide")

st.write("**By: [Bilal Mussa](https://www.linkedin.com/in/bilalmussa/)**")


@st.cache_data
def tidy_data(data):
    # clean up some of the data where needed
    data["date"] = pd.to_datetime(data["date"]).dt.date
    data = data.fillna(0)
    return data


st.sidebar.write("Please load your data in")

user_input = st.sidebar.file_uploader("Upload CSV", type=["csv"])

if user_input is None:
    st.stop()

raw_data = tidy_data(pd.read_csv(user_input, parse_dates=["date"], dayfirst=True))

model_vars = list(raw_data.columns)
model_vars.remove("date")

st.sidebar.write("Please select your dependant variable")

option = st.sidebar.selectbox("Select your dep var?", (model_vars))

st.sidebar.write("You selected:", option)

if "model_specs" not in st.session_state:
    # Now we will create the model spec page/editable dataframe
    model_specs = pd.DataFrame(model_vars, columns=["variable"])
    model_specs["log"] = 0
    model_specs["Adstock"] = 0
    model_specs["DimRet"] = 0
    model_specs["Coef."] = 0
    model_specs["Std.Err."] = 0
    model_specs["t"] = 0
    model_specs["P>|t|"] = 0
    model_specs["Override"] = 1
    model_specs["Fixed"] = 0
    model_specs["DecompGroup"] = ""
    st.session_state["model_specs"] = model_specs

edited_df = st.experimental_data_editor(st.session_state["model_specs"])
if edited_df.index.name != "variable":
    edited_df = edited_df.set_index("variable")

selected_vars = list(edited_df[edited_df["Override"] == 1].index)

y = raw_data[option]
x = raw_data[selected_vars]

model = sm.OLS(y, x).fit()
predictions = model.predict(x)
LRresult = model.summary2().tables[1]
LRresult = LRresult.rename_axis("variable")

# print_model = model.summary()
# st.text(print_model)
# print(print_model)

if st.button("Update values"):
    edited_df.update(LRresult)
    st.session_state["model_specs"] = edited_df
    st.experimental_rerun()

[bilalmussa89]

Thank you for this. I can confirm that it works exactly as what I had in mind. I didnt know about session state so something new to learn.