2026-06-09 –, Maschinenhaus
For a golden set, you need queries. Even if you have them, you can’t judge all docs for each query. Only the top N. How do we rank the top N? See the circular dependency? We’ll talk about ways to untangle it: lexical search, significant terms, training an embedder from scratch, etc. By iteratively refining data and queries, we'll get there.
If you’re not satisfied with your golden set or don’t have it at all, this session is for you. You may have queries (e.g., from query logs) or you need to generate them. We’ll start by looking at how to create synthetic queries from individual documents, as well as from facets and facet combinations, that might match N documents.
We’ll move on to relevance judgements. Even with LLM-as-a-judge, it’s not feasible to, say, rate a 1M doc corpus for 1K queries. We need the top N. How do we know the "correct" top N? We’ll need to explore the dataset for any query that is ambiguous (i.e., doesn't clearly match a single doc). There are different methods for exploring data: visualizations, analysis tweaks (e.g., stemming, synonyms)... Vector similarity also helps, but choosing an embedder is tricky because transfer learning can introduce bias that may be misleading for our dataset.
We can’t get a perfect golden set on the first try, but we’ll explore techniques to iterate until we’re happy. Which is important for any new search application, whether it’s central to the business (i.e., larger teams, bigger budget) or not.
Radu has been in the search space for many years, mainly on Elasticsearch, Solr, OpenSearch, and, more recently, Vespa.ai. Helps users with both the relevance and the operations side of retrieval. Enjoys education in all its forms (training, blog posts, books, conferences...) and got the chance to be involved in all of them.
Author, software engineer, trainer and consultant focused on information retrieval. In his work helping companies throughout the whole software lifecycle - from requirements gathering and architecture, through implementation and deployment ending with scaling and tuning. In his free time a novice carpenter and ultra runner with varying degree of success.