AppTest: test & build interactive Python data apps faster

[jcarroll]

A native framework for automated app testing

By Joshua Carroll

Posted in Product, October 31 2023

Every data app builder wants to build a flawless app in record time, but speed and quality may feel at odds with each other. Imagine pouring your heart into coding a beautiful new Python app, eager to share it with the world. However, before you can deploy, you can’t rush testing, else you’ll risk a code error tarnishing your app experience. 

We've heard your pain and felt this ourselves. In practice, you probably do some manual sanity tests of your app changes and hope for the best. Yesterday’s automated test options are usually complex and hard to maintain:

• Conduct unit tests on the backend logic by factoring it out from the UI
• Set up a heavyweight browser testing framework like Selenium, Playwright, or Cypress for end-to-end testing

Luckily, you don’t have to live in this reality anymore. You can develop faster and ensure high quality!

Introducing: AppTest

AppTest is a new automated way to write and execute tests natively in Streamlit. Developers can use this API to confirm that all aspects of their app are working correctly. 

With this automated testing framework you can: 

Code with confidence: Run all your tests with a single command with Pytest. You no longer need to factor out your unit testable code or do extensive manual testing. Dealing with heavy end-to-end testing frameworks can be a thing of the past. 

Collaborate seamlessly: Build apps with your team without worrying about breaking existing workflows. By connecting Streamlit to tools like GitHub Actions you can build a continuous integration pipeline that automatically runs tests after each commit.

Feel more comfortable with complexity. Go beyond prototypes and build more powerful apps to take your data apps to the next level.

Simple, powerful, and all in Python. 💪

How AppTest works 

Now you can test each feature, interaction, or app logic headlessly via API. By headlessly, we mean that you can test the result without having to preview in the browser.  You can use the API reference docs to build out different scenarios you want to test. 

When you are ready, test everything with Pytest, locally and/or with GitHub Actions. View the results that will confirm that your features are all working correctly (or not). 

AppTest in action 

Watch the video to take a tour of AppTest. In the demo video will cover: 

Ready, set, test! 

Start building and executing tests faster with AppTest to have more control over your app experiences. Check out the docs to get started. 

Show off what you have built! 

Share a link to tests you built for your Community Cloud app and show them successfully running in GitHub Actions. 

You can share an example by posting a link to your test file like this, and then share a successful run in a link like this. To get started setting up GitHub Actions, take a look at GitHub’s tutorial or use our sample workflow file.

If you provide your email in the comment with the two links, we will send the first 10 examples a Streamlit t-shirt! 

Happy Streamlit-ing 🎈

---

This is a companion discussion topic for the original entry at https://blog.streamlit.io/apptest-faster-automated-testing

[fabiogra]

Cool, this is something really appreciated!
Now I can finally have a proper method to test streamlit apps.

I tried AppTest in Moseca and it’s green light

Here’s test file (I added also a conftest file for fixtures) and the github workflow.

[digitalghost-dev]

I seem to be getting RunTime errors with my testing:

RuntimeError: AppTest script run timed out after 3s)

venv/lib/python3.11/site-packages/streamlit/testing/v1/local_script_runner.py:145: RuntimeError
============================ short test summary info ============================
FAILED tests/test_streamlit_app.py::test_app - RuntimeError: AppTest script run timed out after 3s)

Trying to read the docs but not seeing anything on this issue.

[jcarroll]

Hi @digitalghost-dev you can see from the error message that the issue is timeout. You can fix it by setting a longer timeout that makes sense for your app run. Here’s a quick example:

from streamlit.testing.v1 import AppTest

def test_example():
    # set default_timeout on init
    at = AppTest.from_file("app.py", default_timeout=30)

    # OR set timeout on each run
    at.run(timeout=30)

I hope it helps. You can read more in the docs.

[atifumacaxi]

Waiting to know if it worked @digitalghost-dev !
cheers!

[digitalghost-dev]

Yes, it worked. Getting another error but not related to the RunTimeError I was getting from my original question! @atifumacaxi

[digitalghost-dev]

Thank you.

[digitalghost-dev]

Hey, @jcarroll, I have another question for you with testing.

I’m wanting to implement this testing feature with GItHub Actions and ran into an issue.

Context

A part of my Streamlit dashboard uses MapBox and I store the key inside of .streamlit/secrets.toml. When I run pytest locally, everything is fine but when I deploy this on GitHub Actions, my workflow fails. I found the issue to be that part of my code can’t access .streamlit/secrets.toml because,well, it’s not committed to the repository so the test fails! I tested this by removing secrets.toml from local repository and it failed, just like the CI deployment.

Using the pytest-cov plugin, the error start on the same lines, locally and on GitHub Actions.

Solution?

So, I am not sure how to handle this caveat. I do know that there is this section in the docs but the examples don’t use secrets so not sure if it’s the same. If you have any ideas, that would be helpful!

[jcarroll]

Hi @digitalghost-dev, two options (I think):

• If you want to run the mapbox calls in your CI, you can set a secret value in the test using AppTest.secrets["MAPBOX_TOKEN"] = "{token}" (or whatever the right key is). Obviously you don’t want to commit the actual token in your test. You could load the secret from an environment variable and then set it in your workflow as described here.
• If you don’t need to test the actual mapbox call in your CI, a common approach would be to mock that call in the test using unittest.patch. You can see an example doing this for a langchain OpenAI API call here.

These would be the general approaches to use for API calls. I’m not sure if it’s exactly the right recommendation for your use case. If you want to provide a fuller example (such as example code or pointer to a repo) I’d be happy to take a closer look. It might make sense to start a new topic for that discussion and just tag me there Hope this is helpful! Cheers

Edit: For any other interested readers, find the longer discussion and solution for this question here

[lee_kah_how]

Nice, went ahead to try for a super basic example using AppTest here

Here is the test file and the successful run of github workflow.

Glad to get it to run! (: Will be integrating this for more complex products that I have in production

[digitalghost-dev]

Finally got my tests to run locally and on GitHub Actions. I actually went with a keyless authentication method using this GitHub Action: google-github-actions/auth@v1 (which I use in other sections of my CI pipeline).

Here are the local tests:

Here is the completed job implemented with my current CI pipeline!

Here is the test file.

Email: streamlit.antonym268@passmail.net

[jcarroll]

Awesome! Congrats!! It looks great!

[heitormarcal]

Hi!
Does the file I want to test need to be at the same folder where I’m running the test file?

For example, let’s say I have the following:
myproject/
├── app.py
└── tests/
└── test_app.py

when I try to run my app.py from test_app.py, it seems to try to access app.py from “tests” directory.

Thanks!