Memory consumption issues

🎈 Using Streamlit
,
8

Hi @Jonathan_Rhone.

Thanks for yuor replies, they have been very useful!

Ok, just to be sure that we have the same information, displaying the same values and having the same code, I drop here the latest code i used. It uses the exact same data set I shared with you before.

With this code, and this sample of data, the script runs, displays the information and I manage to work with the options in the browser. I made a small check on the memory consumption, and as long as I use this data set, things work fine more or less, not too fast, not too slow.

The tool I use to check the memory of my laptop is a basic “htop” in the console. While using this data set you have, memory is ok. However, if I move to use a large data set, this is what I get:

Captura de pantalla de 2020-04-03 17-35-57
Captura de pantalla de 2020-04-03 17-35-571274×840 257 KB

The memory and the swap are full, and it seems that chrome is the one taking all of it (or at least a great decent part). Plus, the page in chrome crashes. I tryied to check for the tools you mentioned, but I haven’t found yet the same information in the dev tools of chrome. I will keep looking!

Here you see the code, so you can just run it:

import pandas as pd
import os
import numpy as np
import datetime as dt
import glob
import calendar 
import pdb
import streamlit as st
import matplotlib
import matplotlib.pyplot as plt
import plotly.graph_objs as go
import plotly.express as px


st.title('APP Oriente')

# READ THE DATA FROM FILE === NOT STORED IN SERVER ... YET
@st.cache
def load_data():
    files_only = glob.glob('*.csv')

    data = pd.DataFrame()
    for i in files_only: 
        df = pd.read_csv(i) 
        data = data.append(df)
    
    # Sort data by date
    data.sort_values(by='Tiempo Sistema', inplace=True, ascending=True)

    # Drop columns
    cols_to_drop=['Bar Trend', 'Next Record',
       'Inside Temperature', 'Inside Humidity',
       '10 Min Avg Wind Speed', 'Extra Temperatures', 'Soil Temperatures', 'Leaf Temperatures',
       'Extra Humidties', 'Storm Rain', 'Start Date of current Storm',
       'Month Rain', 'Year Rain', 'Day ET', 'Month ET', 'Year ET',
       'Soil Moistures', 'Leaf Wetnesses', 'Inside Alarms', 'Rain Alarms',
       'Outside Alarms', 'Extra Temp/Hum Alarms', 'Soil & Leaf Alarms',
       'Transmitter Battery Status', 'Console Battery Voltage',
       'Forecast Icons', 'Forecast Rule number', 'Time of Sunrise',
       'Time of Sunset', '<LF> = 0x0A', '<CR> = 0x0D', 'CRC']
    data.drop(cols_to_drop, axis=1, inplace = True)

    index = np.arange(0,len(data))
    data = data.set_index(index) 

    # Apply basic transformations
    data['Tiempo Sistema'] = pd.to_datetime(data['Tiempo Sistema']) - pd.to_timedelta(5,unit='h')
    data['Rain Rate'] = data['Rain Rate']*0.2/10. #in units of cm/hour
    data['Barometer'] = data['Barometer']/1000. + 760 # FALTA CORREGIR ESTE CLALCULO
    data['Outside Temperature'] = ( data['Outside Temperature']/10. - 32.) * (5.0/9.0)
    data.loc[(data['Outside Temperature'] > 50) | (data['Outside Temperature'] < -15)] = np.nan
    
    return data

# Load de data
df = load_data()

st.title('Última actualización')

'## Fecha[Y-M-D]/hora',df['Tiempo Sistema'].iloc[-1]
'## Temperatura',round(df['Outside Temperature'].iloc[-1],2),'°C'
'## Presión',df['Barometer'].iloc[-1],'hPa'

st.title('Gráficos')

st.header("Plotly Temperatura")
min_date_data = pd.to_datetime(df['Tiempo Sistema']).min()
max_date_data = pd.to_datetime(df['Tiempo Sistema']).max()

st.sidebar.markdown('# Fechas plots')
tmp1 = st.sidebar.date_input('Fecha inicial',min_date_data)
tmp2 = st.sidebar.date_input('Fecha final')

ok = ( (pd.to_datetime(df['Tiempo Sistema']) >= pd.to_datetime(tmp1)) & \
       (pd.to_datetime(df['Tiempo Sistema']) <  pd.to_datetime(tmp2)) )

st.write('Gráfico de temperatura entre',tmp1,' y ',tmp2)
df2 = df[ok]
time_temp_plt = go.Scatter(x=df2['Tiempo Sistema'], y=df2['Outside Temperature'], mode = 'markers')
df_plt = [time_temp_plt]
layout = go.Layout(xaxis_title="Date", yaxis_title="Temperature [C]")
fig = go.Figure(data=df_plt, layout=layout)
st.write(fig)

st.header("Plotly Presión")
st.write('Gráfico de presión entre',tmp1,' y ',tmp2)
time_temp_plt = go.Scatter(x=df2['Tiempo Sistema'], y=df2['Barometer'], mode = 'markers')
df_plt = [time_temp_plt]
layout = go.Layout(xaxis_title="Date", yaxis_title="Presión [hpa]")
fig = go.Figure(data=df_plt, layout=layout)
st.write(fig)

if st.checkbox('Mostrar datos'):
    '## Data',len(df2)
    df2

Thanks again!