Searching large data sets in (near) constant time
06-20, 14:50–15:30 (Europe/Berlin), Frannz Salon

Tackle large search results by estimating hit count, interpolating a first phase ranking and limiting
the returned result set to the most relevant documents in a multi-million document index.

In low latency search environments, queries producing large result sets are a real pain. A proper ranking of large result sets burns a lot cpu. Those queries have the potential to slow down or even brick your cluster. On the customer side it is questionable whether it makes sense to return millions of documents as the customer has to filter them afterwards anyway.

Those large result sets caused us heavy headache as they significantly reduced the available compute head room on the nodes of our Solr cluster. They even bricked the whole cluster when hitting the cluster in high volume. In this project report we'll guide you through the steps (and math) how we:

  • constructed index based random experiments,
  • estimate the rough query hit count of a query by extrapolating bucket search results,
  • collect and apply static first phase ranking information,
  • use the information collected to filter the result set to the most relevant documents to return no more than a given number of documents,
  • extrapolate hit and facet counts to mimic the original search result and
  • handle document collapsing and facetting.

In this talk we'll guide you through the software architectural aspects as well as the math applied. Although applied on a Solr search system, this concept can be applied on other search engines as well.

See also: Slides (5.4 MB)

Torsten is a freelance search & operations engineer with a focus on open-source search, container, and cloud technology. He tweaks Apache Solr installation in the cloud and on bare-metal with a focus on observability.

Dennis Berger is a freelance software and infrastructure engineer. He started his career in the industry developing low-latency applications and infrastructures with deep knowledge of the operating system, kernel, and application code. Now he focuses on developing fast and resource-efficient applications across the stack, from the IO path in the kernel to the user space, using innovative and modern technologies.