A 3-step guide on collecting, analyzing, and managing AI model feedback
Hey, community! 👋
My name is Jeff, and I’m a co-founder of Trubrics. As a data scientist and Machine Learning engineer, I have experienced firsthand the challenges of deploying new ML models without understanding how users interact with them. This can lead to reduced model performance and, ultimately, misaligned models and users.
More specifically, here is why you should start listening to your users:
- 🚨 Identify bugs: Users constantly run inference on your models and may be more likely to find bugs than your ML monitoring system.
- 🧑💻️ Fine-tune: Users often have domain knowledge that can be useful in fine-tuning models.
- 👥 Align: Identifying user preferences can help align models with users.
If you want to generate user insights on your AI models, we've built Trubrics, the first user insights platform for AI models.
In this post, we'll cover:
- How to create a free account with Trubrics
- How to collect user feedback from your AI Streamlit app
- How to analyze and manage your user feedback in Trubrics
1. How to create a free account with Trubrics
Create an account directly in the Trubrics app:
After logging in, you'll find a pre-existing
Click on it to access code snippets and start collecting feedback. You can create different feedback components to collect and organize feedback across multiple projects or apps.
Now, to save your first piece of feedback to the default component, head over to our example user feedback LLM app:
2. How to collect user feedback from your AI Streamlit app
Let's explore various code snippets to see how you can embed Trubrics feedback components directly into your app.
First, install our SDK with the
streamlit dependency (if it's not already installed):
pip install "trubrics[streamlit]"
Then copy and paste this snippet into your app:
import streamlit as st from trubrics.integrations.streamlit import FeedbackCollector collector = FeedbackCollector( component_name="default", email=st.secrets["TRUBRICS_EMAIL"], password=st.secrets["TRUBRICS_PASSWORD"], ) collector.st_feedback( feedback_type="thumbs", model="your_model_name", open_feedback_label="[Optional] Provide additional feedback", )
What's going on here? Let's break it down:
FeedbackCollectorobject. Store your Trubrics credentials in st.secrets, and specify the feedback component to which you want to save. In this case, use the
st_feedback()method. Calling this method allows users to embed UI widgets in their apps. These widgets can be added throughout your app to collect feedback on different predictions.
And that's it!
Now you'll see this feedback widget in your application:
3. How to analyze and manage your user feedback in Trubrics
After you've saved the feedback to Trubrics, consult the 🪄 Insights page for quantitative and qualitative analysis. This enables AI teams to understand whether users are satisfied with predictions and compare results between different models.
Various filters on the
User feedback tab allow AI teams to:
- Aggregate responses by frequency (hourly, daily, weekly, monthly)
- View all responses for a specific score, model, or user
- Compare responses for all scores, models, or users
For qualitative feedback, user comments are collected in the
text field of the Feedback response. All comments are listed in the
Comments tab and may be grouped to create an issue. AI model issues can also be opened or closed on the issues page.
Thank you for reading! You've learned how to collect app user feedback and view it in Trubrics. If you have any questions, please post them in the comments below or contact me directly on my LinkedIn.
Happy Streamlit-ing! 🎈
This is a companion discussion topic for the original entry at https://blog.streamlit.io/trubrics-a-user-feedback-tool-for-your-ai-streamlit-apps