2025-06-17 –, Maschinenhaus
Robust Search Evaluation is both a “must have” for any modern day Search team and an “after thought” that never gets the team’s full attention. This is especially true with the various open source search engines. Most teams build their own data collection and eval tools. Some use standalone open source tools. We present a better solution!
In this talk we will lay out the history of Search Evaluation, why it’s critical in today’s AI powered world, and make the case for why Search Evaluation needs to be part and parcel of any modern Search Engine. We will share some lessons from building multiple Search Evaluation toolsets, including the popular open source tool Quepid, and why we felt we needed to build the Search Relevance Workbench as an integrated suite. We will show how SRW collects user click behavior using the User Behavior Insights open standard, and how click data is combined with labeled data to measure search quality. We’ll show how you can use that information to run optimizers like Learning to Boost and Hybrid Search Optimizers that replace traditional manually tuned algorithms. You will leave understanding how SRW is different from previous tools, and how you can take advantage of it with your own search engine (not just OpenSearch) as well.
Search, Data Science
Level:Intermediate
Stavros Macrakis is the senior technical product manager for OpenSearch focusing on document and e-commerce search. He has worked on search for 20 years and is passionate about search relevance.
Eric Pugh is the co-founder of OpenSource Connections. Today he helps OSC’s clients, especially those in the ecommerce space, build their own search teams and improve their search maturity, both by leading projects and by acting as a trusted advisor.
He is an active maintainer on the OpenSearch Documentation project, and is focused on expanding the suite of Search Relevance features in the OpenSearch Project.
Fascinated by the craft of software development, Eric Pugh has been involved in the open source world as a tester, developer, committer and user for the past twenty years. He is a member of the Apache Software Foundation and co-authored the book Apache Solr Enterprise Search Server, now on its third edition.
OpenSource Connections mission to empower the world’s search teams comes directly from Eric’s belief in the open source software movement, and the importance of educating people to succeed with it, so that people own their technology.
When not thinking about search, Eric likes to get his hands dirty by building furniture. His next project is a reproduction Danish modern couch, using just hand tools!