How to build a real-time live dashboard with Streamlit

Show the Community!
1

5 easy steps to make your own data dashboard

Posted in Community, April 21 2022

Ever thought you could build a real-time dashboard in Python without writing a single line of HTML, CSS, or Javascript?

Yes, you can! In this post, you’ll learn:

  1. How to import the required libraries and read input data
  2. How to do a basic dashboard setup
  3. How to design a user interface
  4. How to refresh the dashboard for real-time or live data feed
  5. How to auto-update components

Can’t wait and want to jump right in? Here's the code repo and the video tutorial.

What’s a real-time live dashboard?

A real-time live dashboard is a web app used to display Key Performance Indicators (KPIs).

If you want to build a dashboard to monitor the stock market, IoT Sensor Data, AI Model Training, or anything else with streaming data, then this tutorial is for you.

1. How to import the required libraries and read input data

Here are the libraries that you’ll need for this dashboard:

  • Streamlit (st). As you might’ve guessed, you’ll be using Streamlit for building the web app/dashboard.
  • Time, NumPy (np). Because you don’t have a data source, you’ll need to simulate a live data feed. Use NumPy to generate data and make it live (looped) with the Time library (unless you already have a live data feed).
  • Pandas (pd). You’ll use pandas to read the input data source. In this case, you’ll use a Comma Separated Values (CSV) file.

Go ahead and import all the required libraries:

import time  # to simulate a real time data, time loop
import numpy as np  # np mean, np random
import pandas as pd  # read csv, df manipulation
import plotly.express as px  # interactive charts
import streamlit as st  # 🎈 data web app development

You can read your input data in a CSV by using pd.read_csv(). But remember, this data source could be streaming from an API, a JSON or an XML object, or even a CSV that gets updated at regular intervals.

Next, add the pd.read_csv() call within a new function get_data() so that it gets properly cached.

What's caching? It's simple. Adding the decorator @st.experimental_memo will make the function get_data() run once. Then every time you rerun your app, the data will stay memoized! This way you can avoid downloading the dataset again and again. Read more about caching in Streamlit docs.

dataset_url = "https://raw.githubusercontent.com/Lexie88rus/bank-marketing-analysis/master/bank.csv"
# read csv from a URL
@st.experimental_memo
def get_data() -> pd.DataFrame:
    return pd.read_csv(dataset_url)
df = get_data()

2. How to do a basic dashboard setup

Now let’s set up a basic dashboard. Use st.set_page_config() with parameters serving the following purpose:

  • The web app title page_title in the HTML tag <title> and in the browser tab
  • The favicon that uses the argument page_icon (also in the browser tab)
  • The layout = "wide" that renders the web app/dashboard with a wide-screen layout
st.set_page_config(
    page_title="Real-Time Data Science Dashboard",
    page_icon="✅",
    layout="wide",
)

3. How to design a user interface

A typical dashboard contains the following basic UI design components:

  • A page title
  • A top-level filter
  • KPIs/summary cards
  • Interactive charts
  • A data table

Let’s drill into them in detail.

Page title

The title is rendered as the <h1> tag. To display the title, use st.title(). It’ll take the string “Real-Time / Live Data Science Dashboard” and display it in the Page Title.

# dashboard title
st.title("Real-Time / Live Data Science Dashboard")

Top-level filter

First, create the filter by using st.selectbox(). It’ll display a dropdown with a list of options. To generate it, take the unique elements of the job column from the dataframe df. The selected item is saved in an object named job_filter:

# top-level filters
job_filter = st.selectbox("Select the Job", pd.unique(df["job"]))

Now that your filter UI is ready, use job_filter to filter your dataframe df.

# dataframe filter
df = df[df["job"] == job_filter]

KPIs/summary cards

Before you can design your KPIs, divide your layout into a 3 column layout by using st.columns(3). The three columns are kpi1, kpi2, and kpi3. st.metric() helps you create a KPI card. Use it to fill one KPI in each of those columns.

st.metric()’s label helps you display the KPI title. The value **is the argument that helps you show the actual metric (value) and add-ons like delta to compare the KPI value with the KPI goal.

# create three columns
kpi1, kpi2, kpi3 = st.columns(3)
# fill in those three columns with respective metrics or KPIs
kpi1.metric(
    label="Age ⏳",
    value=round(avg_age),
    delta=round(avg_age) - 10,
)
kpi2.metric(
    label="Married Count 💍",
    value=int(count_married),
    delta=-10 + count_married,
)
kpi3.metric(
    label="A/C Balance ＄",
    value=f"$ {round(balance,2)} ",
    delta=-round(balance / count_married) * 100,
)

Interactive charts

Split your layout into 2 columns and fill them with charts. Unlike the metric above, use the with clause to fill the interactive charts in the respective columns:

  • Density_heatmap in fig_col1
  • Histogram in fig_col2
# create two columns for charts
fig_col1, fig_col2 = st.columns(2)
with fig_col1:
    st.markdown("### First Chart")
    fig = px.density_heatmap(
        data_frame=df, y="age_new", x="marital"
    )
    st.write(fig)
   
with fig_col2:
    st.markdown("### Second Chart")
    fig2 = px.histogram(data_frame=df, x="age_new")
    st.write(fig2)

Data table

Use st.dataframe() to display the data frame. Remember, your data frame gets filtered based on the filter option selected at the top:

st.markdown("### Detailed Data View")
st.dataframe(df)

4. How to refresh the dashboard for real-time or live data feed

Since you don’t have a real-time or live data feed yet, you’re going to simulate your existing data frame (unless you already have a live data feed or real-time data flowing in).

To simulate it, use a for loop from 0 to 200 seconds (as an option, on every iteration you’ll have a second sleep/pause):

for seconds in range(200):
    df["age_new"] = df["age"] * np.random.choice(range(1, 5))
    df["balance_new"] = df["balance"] * np.random.choice(range(1, 5))
    time.sleep(1)

Inside the loop, use NumPy's random.choice to generate a random number between 1 to 5. Use it as a multiplier to randomize the values of age and balance columns that you’ve used for your metrics and charts.

5. How to auto-update components

Now you know how to do a Streamlit web app!

To display the live data feed with auto-updating KPIs/Metrics/Charts, put all these components inside a single-element container using st.empty(). Call it placeholder:

# creating a single-element container.
placeholder = st.empty()

Put your components inside the placeholder by using a with clause. This way you’ll replace them in every iteration of the data update. The code below contains the placeholder.container() along with the UI components you created above:

with placeholder.container():
    # create three columns
    kpi1, kpi2, kpi3 = st.columns(3)
    # fill in those three columns with respective metrics or KPIs
    kpi1.metric(
        label="Age ⏳",
        value=round(avg_age),
        delta=round(avg_age) - 10,
    )
    
    kpi2.metric(
        label="Married Count 💍",
        value=int(count_married),
        delta=-10 + count_married,
    )
    
    kpi3.metric(
        label="A/C Balance ＄",
        value=f"$ {round(balance,2)} ",
        delta=-round(balance / count_married) * 100,
    )
    # create two columns for charts
    fig_col1, fig_col2 = st.columns(2)
    
    with fig_col1:
        st.markdown("### First Chart")
        fig = px.density_heatmap(
            data_frame=df, y="age_new", x="marital"
        )
        st.write(fig)
        
    with fig_col2:
        st.markdown("### Second Chart")
        fig2 = px.histogram(data_frame=df, x="age_new")
        st.write(fig2)
    st.markdown("### Detailed Data View")
    st.dataframe(df)
    time.sleep(1)

And...here is the full code!

import time  # to simulate a real time data, time loop
import numpy as np  # np mean, np random
import pandas as pd  # read csv, df manipulation
import plotly.express as px  # interactive charts
import streamlit as st  # 🎈 data web app development
# read csv from a github repo
dataset_url = "https://raw.githubusercontent.com/Lexie88rus/bank-marketing-analysis/master/bank.csv"
# read csv from a URL
@st.experimental_memo
def get_data() -> pd.DataFrame:
    return pd.read_csv(dataset_url)
df = get_data()
st.set_page_config(
    page_title="Real-Time Data Science Dashboard",
    page_icon="✅",
    layout="wide",
)
# dashboard title
st.title("Real-Time / Live Data Science Dashboard")
# top-level filters
job_filter = st.selectbox("Select the Job", pd.unique(df["job"]))
# creating a single-element container
placeholder = st.empty()
# dataframe filter
df = df[df["job"] == job_filter]
# near real-time / live feed simulation
for seconds in range(200):
    df["age_new"] = df["age"] * np.random.choice(range(1, 5))
    df["balance_new"] = df["balance"] * np.random.choice(range(1, 5))
    # creating KPIs
    avg_age = np.mean(df["age_new"])
    count_married = int(
        df[(df["marital"] == "married")]["marital"].count()
        + np.random.choice(range(1, 30))
    )
    balance = np.mean(df["balance_new"])
    with placeholder.container():
        # create three columns
        kpi1, kpi2, kpi3 = st.columns(3)
        # fill in those three columns with respective metrics or KPIs
        kpi1.metric(
            label="Age ⏳",
            value=round(avg_age),
            delta=round(avg_age) - 10,
        )
        
        kpi2.metric(
            label="Married Count 💍",
            value=int(count_married),
            delta=-10 + count_married,
        )
        
        kpi3.metric(
            label="A/C Balance ＄",
            value=f"$ {round(balance,2)} ",
            delta=-round(balance / count_married) * 100,
        )
        # create two columns for charts
        fig_col1, fig_col2 = st.columns(2)
        with fig_col1:
            st.markdown("### First Chart")
            fig = px.density_heatmap(
                data_frame=df, y="age_new", x="marital"
            )
            st.write(fig)
            
        with fig_col2:
            st.markdown("### Second Chart")
            fig2 = px.histogram(data_frame=df, x="age_new")
            st.write(fig2)
        st.markdown("### Detailed Data View")
        st.dataframe(df)
        time.sleep(1)

To run this dashboard on your local computer:

  1. Save the code as a single monolithic app.py.
  2. Open your Terminal or Command Prompt in the same path where the app.py is stored.
  3. Execute streamlit run app.py for the dashboard to start running on your localhost and the link would be displayed in your Terminal and also opened as a new Tab in your default browser.

Wrapping up

Congratulations! You have learned how to build your own real-time live dashboard with Streamlit. I hope you had fun along the way.

If you have any questions, please leave them below in the comments or reach out to me at 1littlecoder@gmail.com or on Linkedin.

Thank you for reading, and Happy Streamlit-ing! 🎈

This is a companion discussion topic for the original entry at https://blog.streamlit.io/how-to-build-a-real-time-live-dashboard-with-streamlit/
4 Likes
2

the “st.set_page_config” needs to be called before the “df = get_data()”, probably if u try to run this webapp will be an error like "set_page_config() can only be called once per app, and must be called as the first Streamlit command in your script" or something else, otherwise nice app!

1 Like
3

You’re totally right, nice catch! We’ve updated the post with the fixed code.

3 Likes
4

happy to help :slight_smile:

5

You used a for loop throughout the page

If I still need the input component, I won’t get a response.

Is there any way to put a for loop into an asynchronous thread

6

Hey @yifangtao, unfortunately, adding input widgets from an async thread is currently not possible. All input widgets need to be added to your app before you run the for loop. But you can still have input widgets underneath the live updating charts on the frontend by using st.empty. If you move the placeholder = st.empty() above the st.selectbox in the example above, the selectbox widget will be rendered under the live updating charts:

image
image2046×1618 281 KB

This does not solve all use cases, but maybe it helps with your app?

7

how to add search box to the data table created.

8

The dataframe has a built-in search. Unfortunately, it only can be activated by the user via the search hotkey (CMD+F), there isn’t any other way as of now to trigger this search. What you can implement yourself on the Python side is a filter system: 1) add st.text_input 2) manually filter the pandas dataframe based on the user input before calling st.dataframe

9

This solution give you another choice, it can show, search, edit, download dataframe directly.

1 Like
10

I have a CSV file which updates regularly. However I cannot get the streamlit app to update the plots. My code is exactly same with the following exceptions. I just get an initial plot with data that was present when running the app for the first time. Any help/suggestion would be great.

# Path to the data file
datafile = path.join('data','output.csv')

# Read the data CSV file 
@st.experimental_memo
def get_data() -> pd.DataFrame:
    return pd.read_csv(datafile)



placeholder = st.empty()

while True:
    df = get_data()
    df['time'] = pd.to_datetime(df['time'])
    with placeholder.container():
            # columns for charts
           fig_col1, fig_col2 = st.columns(2)
   ...... # Code here plotting the graphs
           time.sleep(2)
11

Hi @mabhijithn

Before going all the way to plotting your data, just try and load only the dataframe to the screen using the st.dataframe. You should see the dataframe being revised incrementally onscreen after the set time intervals. If this does not work, your plotting will not, too.

If the dataframe gets revised to the screen but the plotting still does not work, try:
a. removing the @st.experimental_memo, to see if the un@@#
b. forcing a page rerun after your sleep command, using: st.experimental_rerun()

Cheers

1 Like
12

Hi @Shawn_Pereira ,

Thanks a lot for the comments! Removing @st.experimental_memo was sufficient to make it work, yes. However, that would mean the get_data is reading data each time. Is there a way to implement caching somehow?

13

You have a workaround for that.

  1. Use python os functions and store the file date/time into a session variable
  2. If this variable has changed during a given time moment of the while loop, then read the dataframe from disk and plot thereafter, otherwise, read in the next iteration of the time sleep function

You could also go through the streamlit caching docs for further detailed info.

Cheers

2 Likes
14

If I do a while True: to always update. when will the loop end?
I would expect the loop to break when the user leaves the page, is streamlit interrupt the python thread running in the backgroud?

1 Like
15

Hi, coming to this issue 8 months later…

I have tried to modify the example in the documentation to add a counter button. I just cannot figure out why the counter increments for the first button click and then nothing… Any thoughts?

  • The counter does not increment after the first click in this real time app
  • The counter increments properly in a static app
  • Streamlit, version 1.23.1
  • Python 3.10.9 (main, Mar 8 2023, 10:47:38) [GCC 11.2.0] on linux

import time  # to simulate a real time data, time loop
import numpy as np  # np mean, np random
import pandas as pd  # read csv, df manipulation
import plotly.express as px  # interactive charts
import streamlit as st  # 🎈 data web app development
from datetime import datetime  # to check  time when things update

st.set_page_config(
    page_title="Real-Time Data Science Dashboard",
    page_icon="✅",
    layout="wide",
)

# read csv from a github repo
dataset_url = "https://raw.githubusercontent.com/Lexie88rus/bank-marketing-analysis/master/bank.csv"

# read csv from a URL
@st.cache_data
def get_data() -> pd.DataFrame:
    return pd.read_csv(dataset_url)

df = get_data()

def count_fn():     ## ADDED   
    if 'counter' not in st.session_state:
        st.session_state['message'] = f"counter not in st.session_state {datetime.now()}"
        st.session_state['counter'] = 0
        
    def handle_click():
        st.session_state['counter'] += 1 
    st.button('increment', on_click = handle_click, key = 'button')  


# dashboard title
st.title("Real-Time / Live Data Science Dashboard")

# top-level filters and interactive widgets 
count_fn()
if 'message' in st.session_state: 
    st.write(st.session_state['message'])

job_filter = st.selectbox("Select the Job", pd.unique(df["job"]))

# creating a single-element container
placeholder_counter = st.empty()
placeholder = st.empty()

# dataframe filter
df = df[df["job"] == job_filter]

# near real-time / live feed simulation
for seconds in range(200):

    df["age_new"] = df["age"] * np.random.choice(range(1, 5))
    df["balance_new"] = df["balance"] * np.random.choice(range(1, 5))

    # creating KPIs
    avg_age = np.mean(df["age_new"])

    count_married = int(
        df[(df["marital"] == "married")]["marital"].count()
        + np.random.choice(range(1, 30))
    )

    balance = np.mean(df["balance_new"])

    with placeholder.container():

        # create three columns
        kpi1, kpi2, kpi3 = st.columns(3)

        # fill in those three columns with respective metrics or KPIs
        kpi1.metric(
            label="Age ⏳",
            value=round(avg_age),
            delta=round(avg_age) - 10,
        )
        
        kpi2.metric(
            label="Married Count 💍",
            value=int(count_married),
            delta=-10 + count_married,
        )
        
        kpi3.metric(
            label="A/C Balance ＄",
            value=f"$ {round(balance,2)} ",
            delta=-round(balance / count_married) * 100,
        )

        # create two columns for charts
        fig_col1, fig_col2 = st.columns(2)
        with fig_col1:
            st.markdown("### First Chart")
            fig = px.density_heatmap(
                data_frame=df, y="age_new", x="marital"
            )
            st.write(fig)
            
        with fig_col2:
            st.markdown("### Second Chart")
            fig2 = px.histogram(data_frame=df, x="age_new")
            st.write(fig2)

        st.markdown("### Detailed Data View")
        st.dataframe(df)
        
    with placeholder_counter.container():  # ADDED CODE
        st.metric('COUNTER', st.session_state.counter)
        
    
    time.sleep(1)
16

I wouldn’t enclose a button and it’s on_click callback handler within a function. In theory the callback will not be in scope outside that function. You also seem to have a superfluous ‘.’ in your datetime import.

By using print statements I saw that the button’s on_click wasn’t being called for some reason. Hence the counter state wasn’t being incremented.

This works (note the code is pulled out of the encapsulating function you had):

if 'counter' not in st.session_state:
    st.session_state['counter'] = 0
    
if st.button('increment', key = 'button'):
    st.session_state['counter'] += 1
17

Thanks @asehmi,

  • “You also seem to have a superfluous ‘.’ in your datetime import” : gone… thanks (that was a result of adding the comment for the import in here)

  • "In theory the callback will not be in scope outside that function. " : Your solution works, i.e., removing the function and removing the handler in order to use the increment as you suggest.

However, if we try to use the handler outside the function, the on_click is never called. I think using handlers is important for real time apps… it would be very helpful to understand this issue.

I include the modified example including the handler below…

Thanks
M

import time  # to simulate a real time data, time loop
import numpy as np  # np mean, np random
import pandas as pd  # read csv, df manipulation
import plotly.express as px  # interactive charts
import streamlit as st  # 🎈 data web app development
from datetime import datetime  # to check  time when things update

st.set_page_config(
    page_title="Real-Time Data Science Dashboard",
    page_icon="✅",
    layout="wide",
)

# read csv from a github repo
dataset_url = "https://raw.githubusercontent.com/Lexie88rus/bank-marketing-analysis/master/bank.csv"

# read csv from a URL
@st.cache_data
def get_data() -> pd.DataFrame:
    return pd.read_csv(dataset_url)

df = get_data()

if 'counter' not in st.session_state:
    st.session_state['counter'] = 0
def handle_click():
    st.session_state['counter'] += 1 
st.button('increment', key = 'button', on_click=handle_click)
    
# dashboard title
st.title("Real-Time / Live Data Science Dashboard")

# top-level filters and interactive widgets    
job_filter = st.selectbox("Select the Job", pd.unique(df["job"]))

# creating a single-element container
placeholder_counter = st.empty()
placeholder = st.empty()

# dataframe filter
df = df[df["job"] == job_filter]

# near real-time / live feed simulation
for seconds in range(200):

    df["age_new"] = df["age"] * np.random.choice(range(1, 5))
    df["balance_new"] = df["balance"] * np.random.choice(range(1, 5))

    # creating KPIs
    avg_age = np.mean(df["age_new"])

    count_married = int(
        df[(df["marital"] == "married")]["marital"].count()
        + np.random.choice(range(1, 30))
    )

    balance = np.mean(df["balance_new"])

    with placeholder.container():

        # create three columns
        kpi1, kpi2, kpi3 = st.columns(3)

        # fill in those three columns with respective metrics or KPIs
        kpi1.metric(
            label="Age ⏳",
            value=round(avg_age),
            delta=round(avg_age) - 10,
        )
        
        kpi2.metric(
            label="Married Count 💍",
            value=int(count_married),
            delta=-10 + count_married,
        )
        
        kpi3.metric(
            label="A/C Balance ＄",
            value=f"$ {round(balance,2)} ",
            delta=-round(balance / count_married) * 100,
        )

        # create two columns for charts
        fig_col1, fig_col2 = st.columns(2)
        with fig_col1:
            st.markdown("### First Chart")
            fig = px.density_heatmap(
                data_frame=df, y="age_new", x="marital"
            )
            st.write(fig)
            
        with fig_col2:
            st.markdown("### Second Chart")
            fig2 = px.histogram(data_frame=df, x="age_new")
            st.write(fig2)

        st.markdown("### Detailed Data View")
        st.dataframe(df)
        
    with placeholder_counter.container():  # ADDED CODE
        st.metric('COUNTER', st.session_state.counter)
        
    
    time.sleep(1)
18

The button on_click handler doesn’t appear to do what we expect. This is equivalent and works:

if st.button('increment', key = 'button'):
    handle_click()

I suggest you raise an issue in Streamlit’s GitHub.

19

The GitHub issue opened here

1 Like
20

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.