Judging search relevance seems straightforward: the higher a relevant product ranks, the better your search system works. But when it comes to RAG, things get complicated—there’s no ranking, no traditional documents, just an LLM-generated response to a query. So how do you know if it’s any good? Is there an objective way to measure progress, or are you just guessing?

In this talk, we’ll share a real (if not exactly glamorous) case study of building and evaluating a production RAG system for a fintech company. We’ll cover the headaches of working with a small and noisy corpus, chunking gone wrong, handling low-resource languages (plus users who think your support chatbot is their therapist), and the different frameworks (like RAGAS) to evaluate a RAG system—so you’re not flying blind.