AppTest and Mocking API Without a secrets.toml File Present

Using Streamlit
1

Hey, @jcarroll, first off, I want to thank you for taking the time to review this. I really do appreciate it.

Hopefully I provide enough detail for you.

Context

As mentioned in our previous discussion, I am trying out AppTest so that I can implement native testing with Streamlit. (It’s great that this is a feature by the way).

Issue

The problem I am running into is that I my dashboard uses API keys to reach out to services to provide some functionality. When I first started to write my tests, everything was working fine locally. Then, in a separate test repo on GitHub, I applied the same exact tests but they were failing, After some time, I found out that because the .streamlit/secrets.toml was missing, it couldn’t access the key to continue the tests.

Project Setup

Here is my main repository. The file structure is set where the components/ directory houses certain areas of the dashboard in classes that are then imported to the main streamlit_app.py file. I’m slowly starting to implement classes for organization. Anyway, in the main streamlit_app.py file, I am importing the StadiumMapSection class from components/stadiums_map_section.py and instantiated it on line 18.

Errors

When I run the pytest using this command (locally or not): pytest --cov=streamlit_app tests/ -v , without the .streamlit/secrets.toml being present, the coverage returns:

---------- coverage: platform darwin, python 3.11.6-final-0 ----------
Name               Stmts   Miss  Cover   Missing
------------------------------------------------
streamlit_app.py     229    215     6%   19-726
------------------------------------------------
TOTAL                229    215     6%

So it’s basically skipping the entire file. When I place the the secrets file back in the directory, all the tests pass with no issue.

Here is more output from the failures:

______________________________________________________________________________________ test_title_area ________________________

    def test_title_area():
>       assert "Premier League Statistics / 2023-24" in at.title[0].value

tests/test_streamlit_app.py:7: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = ElementList(), idx = 0

    def __getitem__(self, idx: int | slice) -> El | ElementList[El]:
        if isinstance(idx, slice):
            return ElementList(self._list[idx])
        else:
>           return self._list[idx]
E           IndexError: list index out of range

venv/lib/python3.11/site-packages/streamlit/testing/v1/element_tree.py:229: IndexError
_______________________________________________________________________________________ test_tab_one __________________________

    def test_tab_one():
>       assert at.tabs[0].subheader[0].value == "League Statistics"
E    IndexError: list index out of range

tests/test_streamlit_app.py:12: IndexError

Attempts

I looked at your two suggestions and I tried implementing a mock test with unitest.patch like you mentioned but I could not get it working. (I’ve never used this before).

I tried a couple of variations:

from streamlit.testing.v1 import AppTest
from unittest.mock import patch

at = AppTest.from_file("streamlit_app.py", default_timeout=1000).run()

def test_title_area():
    assert "Premier League Statistics / 2023-24" in at.title[0].value
    assert "Current Round: " in at.subheader[0].value

@patch('streamlit.secrets', return_value={"mapbox": {"mapbox_key": "fake_mapbox_key"}})
def test_tab_one():
    assert at.tabs[0].subheader[0].value == "League Statistics"
    assert at.tabs[0].subheader[1].value == "Current Standings"
    assert at.tabs[0].subheader[2].value == "Location of Stadiums"

and

@patch('components.stadiums_map_section.StadiumMapSection.mapbox_access_token', 'fake_mapbox_key')
def test_tab_one():
    assert at.tabs[0].subheader[0].value == "League Statistics"
    assert at.tabs[0].subheader[1].value == "Current Standings"
    assert at.tabs[0].subheader[2].value == "Location of Stadiums"

but neither worked. I’m not even sure if I am on the right path to be honest.

With more information now, I’m curious to know if you think there is a better option or method of doing this.

2

Hey @digitalghost-dev thanks for the separate post and detailed explanation with links. I took a look through the app code and the hosted app and spent a little time playing around with it. This is cool!! Nice app :slight_smile: Here are a few thoughts and findings that might help you unblock:

Finding issues with tests

I found that printing the AppTest contents is often a good way to figure out what’s going on with the test output when something unexpected is happening. I can slightly modify your first test to do this (note I added some dummy secret values too):

def test_title_area():
	at = AppTest.from_file("streamlit_app.py", default_timeout=1000)
	at.secrets["mapbox"] = {"mapbox_key": "<api key>"}
	at.secrets["gcp_service_account"] = {"client_email": "test@test.com", "token_uri": "some-token-uri"}
	at.run()
	print(at.main)
	assert "Premier League Statistics / 2023-24" in at.title[0].value
	assert "Current Round: " in at.subheader[0].value

Now I run:

pytest -v --disable-warnings -k test_title_area

I can see that the app prints an exception and since I did -v I get the exception printed underneath it too. With this particular test, I see that Google doesn’t like my mocked service account info: google.auth.exceptions.MalformedError: The private_key field was not found in the service account info.

More on that a bit later.

Modular testing

Since you already worked on decomposing your app into components and have several fairly distinct tabs, it might make sense to test those independently. I built a simple test for the StadiumMapSection class.

Notably, I also found that plotly will build a full mapbox chart object even without a real mapbox token, so you can do some running and inspection with a dummy token just fine. Here’s the test I built, which uses AppTest.from_function() to only run the stadium component code and passes:

def test_stadium_map():
	def draw_stadium_map():
		import pandas as pd
		from components.stadiums_map_section import StadiumMapSection
		stms = StadiumMapSection()
		stadium_df = pd.DataFrame({
			"latitude": [51.88],
			"longitude": [-0.42],
			"stadium": ["Kenilworth Road"],
			"team": ["Luton Town"],
		})
		stms.display(stadium_df)
	
	at = AppTest.from_function(draw_stadium_map, default_timeout=1000)
	at.secrets["mapbox"] = {"mapbox_key": "<test api key>"}
	at.run()
	# No exception printed
	assert not at.exception
	# Correct subheader printed
	assert at.subheader[0].value == "Location of Stadiums"

	# Parse out and inspect the chart data (this is still too hacky until we have native support)
	import json
	chart_data = json.loads(at.main.children[1].proto.figure.spec)
	assert chart_data["data"][0]["lat"] == [51.88]
	assert chart_data["data"][0]["hovertext"] == ['Kenilworth Road']

If you liked this approach, I think you could do something similar for the other components.

Easier testing for the whole app - de-composing the monolith query function

As I pointed out above, I ran into trouble mocking your whole app run because of the way the main app file currently tries to load the GCP credentials, then reuses those credentials several places. In particular, I noticed you have the bigquery_connection() function that has 9 return values, mostly dataframes used in the rest of your app.

With a bit of refactoring, I think you could decompose this and make it much easier to build tests for the rest of your app. For example, you could move all the connection building and query logic to its own file with more broken out functions for each dataframe like so:

# components/connections.py

import firebase_admin  # type: ignore
import pandas as pd
import streamlit as st
from firebase_admin import firestore  # type: ignore
from google.cloud import bigquery
from google.oauth2 import service_account  # type: ignore

@st.cache_resource
def firestore_connection():
	credentials = service_account.Credentials.from_service_account_info(
        st.secrets["gcp_service_account"]
    )
	if not firebase_admin._apps:
		firebase_admin.initialize_app()

	return firestore.Client(credentials=credentials)

@st.cache_data(ttl=600)
def run_query(query):
    credentials = service_account.Credentials.from_service_account_info(
        st.secrets["gcp_service_account"]
    )
    query_job = bigquery.Client(credentials=credentials).query(query)
    raw_data = query_job.result()
    data = [dict(data) for data in raw_data]
    return data

@st.cache_resource
def get_standings():
	standings_data = run_query(
		"""
            SELECT rank, logo, team, points, wins, draws, loses, goals_for, goals_against, goal_difference
            FROM `premier_league_dataset.standings`
			ORDER BY rank ASC;
        """
	)
	return pd.DataFrame(data=standings_data)

# ... similar for the other queries

(BTW, I see you followed the tutorial for bigquery connection in Streamlit - I am hoping we can update it soon to use st.connection which might make this a bit easier)

Once you do this update, all the credential handling logic is abstracted out of the main app file and it’s much easier to mock just the database call and expected return value and test your app code against that. Here’s a simple example that assumes the file above exists ^, for the real use case you probably want to run tests of your various tabs and the transforms you do on the dataframes.

# Use patch to mock the return value of get_standings() from above
@patch("components.connections.get_standings")
def test_standings(get_standings):
	def draw_standings():
        # This function is just for demo purpose, for a real test you should build AppTest on some of your app logic :)
		import streamlit as st
		from components.connections import get_standings
		df = get_standings()
		st.dataframe(df)

	# Now you can set a mock return value like this
	get_standings.return_value = pd.DataFrame({
		"team": ["Tottenham", "Manchester"],
		"standings": [5, 2],
	})
	at = AppTest.from_function(draw_standings).run()
    # And test that it gets used by the function
	assert at.dataframe[0].value.iloc[0][0] == "Tottenham"

Alternate approaches / e2e testing

Alternatively, if you want to test that the database and everything is working e2e in your CI, it may make sense to inject the actual GCP credentials as a secret in GitHub Actions and actually run the queries and test against real data in the CI. The limitation in this mocking approach is that it wont catch any mismatches between your actual database schema / ingested data and the app logic.

Setting that up is outside the scope of help I can provide since I’m not an expert, but the starter docs from GitHub are here. If you want to get really crazy, it looks like google supports some keyless approach too (i can’t tell if this totally makes sense for this use case or not).

Hope all this is helpful and makes sense! It was fun and useful for me to dig into this example a bit and understand how to best apply AppTest. Fair warning I may not be able to dig in deep for further follow up questions but if there’s any quick clarification I can provide, I’ll try to do so. Maybe others can jump in to help too. Thanks and good luck!!

1 Like
3

TL;DR

In case you don’t want to do the refactoring above and just want to get e2e tests working with your current approach :wink: Here’s the minimal updates that I think will work:

# tests/test_streamlit_app.py

# ...

# Add a dummy mapbox token secret before running
at = AppTest.from_file("streamlit_app.py", default_timeout=1000)
at.secrets["mapbox"] = {"mapbox_key": "<test api key>"}
at.run()

# streamlit_app.py

# ...

# Load the GCP credentials from an environment variable (JSON string formatted)
# if the expected secret is missing

# Create API client.
if "gcp_service_account" in st.secrets:
	creds = st.secrets["gcp_service_account"]
else:
	import json
	import os
	creds = json.loads(os.environ["GCP_ACCOUNT_STRING"])

credentials = service_account.Credentials.from_service_account_info(creds)

Now, you just need to add the JSON version of service account key file as the environment variable GCP_ACCOUNT_STRING in your GitHub Action workflow from a secret configured for your repository. Loading the secret as an ENV in the workflow should look something like this:

# workflow.yml
# ...
env:
  GCP_ACCOUNT_STRING: ${{ secrets.gcpCredsJSON }}

Note: There are various other ways to set this up that might be cleaner from the various GCP and GitHub documentation but hopefully this is a good quick way :slight_smile:

Since I don’t have access to your DB I couldn’t test this out e2e but I did verify locally that the ENV loading works

1 Like
4

Hey, @jcarroll, thank you so much for writing all of this out! Above and beyond for sure. There’s a lot to think about so I’ll be spending a few days going over your writeup and editing my existing code.

You helped me understand environment variables a bit more and looks like I will need to implement these with my existing CI on GitHub Actions.

Also, thanks for the refactoring advice. Hopefully st.conection() is updated soon for BigQuery usage!

1 Like
5

Hey again, @jcarroll. Just wanted to make a comment that your refactoring of the data connections helped me solve another issue I was having with any sort of input widget! Interacting with an input widget with the old connection setup would kind of freak the page out and seemed to have to re-run everything again and bring me back to the top and with my multiple tabs, it would be bring me back to tab 1. I guess caching the resource helped? Like this issue on the GitHub.

Just tried adding an input widget on tab 5 and it stayed on tab 5! I’ve been dealing with this issue for months lol you saved me.

1 Like
6

Nice, that’s great. It sounds like Component in tab 2 triggers app to jump back to tab 1 · Issue #6257 · streamlit/streamlit · GitHub was the issue you were most likely seeing. We’re talking about how to address that one :slight_smile:

7

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.