🚀 RAGxplorer - Explore the embeddings of your RAG Documents! (GPT 4 + ChromaDB + Sentence Transformers)

💬 Show the Community!
, , , , , ,
1

Meet RAGxplorer - an interactive tool to visualize the retrieval techniques and diagnose which chunks are being retrieved. I hope this application is useful especially for those who are learning RAGs and exploring the embedding space of your documents.

Application

With RAGxplorer, you can:

  1. Upload your own own PDF
  2. Configure the chunk size and chunk overlap
  3. Visualise where their query is in the embedding space, and the chunks that are a top-k match
  4. [NEW!] Experiment with Query Expansion techniques (e.g. multi-questions, hypothetical answer/HyDE)

Screenshot 2024-01-17 at 12.34.05 AM
Screenshot 2024-01-17 at 12.34.05 AM2084×1212 229 KB

Code is here:

Any feedback would be most appreciated!

63 Likes
2

Very cool! How did you store your API key?

2 Likes
3

Thanks!

To clarify, this version is not using any APIs yet. For now, the embedding model is hosted within the streamlit app & the vector db is in memory.

The vector db is chroma - which is great.

on api keys, you may find this blog post useful: 8 tips for securely using API keys

3 Likes
4

Good job!

The course looks very interesting too - thanks for sharing.

4 Likes
5

this is useful! hope to see the “bring your own embeddings” implemented soon!

2 Likes
6

great name, and great app!

3 Likes
7

Coming real soon!

1 Like
8

Does it support word doc?

2 Likes
9

Nope! The app only takes in a PDF (and it has to be a PDF with text, not a scanned text)

10

New update!

I also re-factored the code entirely for more modularity. The code is also now released under MIT License - please feel free to fork it.

1 Like
11

Thanks for sharing your work! :pray:

2 Likes
12

Great repo! the code’s more modular compared to most streamlit apps i’ve seen

2 Likes
13

Screenshot 2024-01-17 at 9.08.12 PM
Screenshot 2024-01-17 at 9.08.12 PM1348×424 22.4 KB

New features - text-embedding-ada-002 and gte-large embedding models!

2 Likes
14

@JacksonChin presently I’m working with my mother language tokenization model for creating my own embeddings

3 Likes
15

Nice! More multilingual tokenizers and language/embedding models will be great = )

2 Likes
16

To share more about what’s going on under the hood, the app uses umap to project these high dimensional embeddings into a 2D space for visualization.

https://umap-learn.readthedocs.io/en/latest/#

17

Once, again thank you to the streamlit team for hosting this competition.

Here’s the repo :package: I hope this helps folks building RAG applications!

For now, the best experience is to clone the repo and run this locally!

The vector database is an in-memory one (Chroma), and I suspect many people using it, or quitting the app while the vector database is being built, causes some issues. Some tweaks to the control flow and deployment may resolve that.

Any feedback for improvement would be most appreciated. I will still continue to work on this.

18

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.