I have a RAG (Retrieval-Augmented Generation) app I’ve built using python, lancedb (oss), and some LLM APIs. lancedb oss is kinda like duckdb in that from my python code, I interact with it like it’s a database, but then there’s no remote db server; it stores the data on my filesystem. unlike duckdb which stores everything in one file, lancedb creates and manages a folder of many files.
I originally interacted with my app (querying certain pdf docs) via my shell (i.e. running python -m my_rag_app --query 'my question for my pdf'). Then I swapped out my command-line UI for a minimal streamlit app, which I can launch successfully locally.
Now I want to deploy it using streamlit cloud, but I’m not sure how I could make that work, given that my data is in lancedb as basically a directory of files. It’s not the kind of data directory that makes sense to check into github, and it’s not a cloud db I can connect to like an API. I think I’m looking for something like a mounted volume in modal (Volumes | Modal Docs).
I’ve looked up streamlit’s supported data sources (Connect to data sources - Streamlit Docs), and this forums page on data persistence (Is there a way to keep data persistent after page reload?), but neither seemed to address my use case.