I’m building a Streamlit-based AI application and experimenting with a separate API gateway for multi-user usage control.
The gateway handles authentication, customer-level quotas, daily limits, usage reporting, and provider-side API credentials. The goal is to keep provider keys out of the frontend while giving each user predictable limits.
I’m interested in feedback on:
Secure secret handling between Streamlit and the gateway
Per-user and per-session quotas
Preventing users from bypassing frontend limits
Tracking streaming responses correctly
Displaying useful usage information without exposing sensitive data
Handling provider failures in a user-friendly way
What patterns have worked well for production Streamlit applications?
Disclosure: I’m building the gateway as part of this project and am looking for implementation feedback.
Thanks for sharing the design—keeping provider credentials behind a gateway is a sensible boundary. Store only the gateway credential in Streamlit secrets, never in UI output or logs; keep .streamlit/secrets.toml out of Git. Streamlit’s Python code runs server-side, but avoid returning credentials in errors or responses. Secrets management · Deployment secrets guidance
For quotas, enforce limits at the gateway—not only in widgets or st.session_state, since those are app/session state, not a substitute for gateway-side enforcement. The sources describe client-side rate limiting as a first defense and hard usage limits as server-side protection, but don’t specify a production quota protocol or how to count streamed usage; make the gateway’s usage accounting authoritative and reconcile it with completed or failed requests. Stream chunks with st.write_stream if appropriate, and show users concise usage totals and actionable failure messages while keeping provider details and credentials private. GenAI app practices Streamlit-specific guidance on customer-level quotas, session attribution, and streaming reconciliation isn’t covered in the available sources.
One case I’d definitely test is the same user sending requests from two browser tabs at once.
If the gateway checks the remaining quota before each request and only updates usage after the response finishes, both requests might pass the check even though their combined cost exceeds the user’s limit.
I’d probably reserve an estimated token budget atomically before starting each request, then adjust it once the actual usage is known.
Streaming makes this a bit trickier. What happens if someone closes the tab halfway through a response, or the provider times out? You wouldn’t want the reservation to stay locked forever, but you also can’t assume the provider charged nothing.
I’d keep that accounting entirely in the gateway, with the user identity coming from verified authentication rather than a Streamlit session ID.
Have you decided how you’ll handle quota reservations and interrupted streams? That seems like one of the easier places for a multi-user app to run into unexpected costs.