2023-06-20 –, Maschinenhaus
Apache Solr’s built-in autoscaling is gone, but the need for autoscaling persists. Using Kubernetes’ HPA, the Solr Operator and new Solr APIs, we re-introduce autoscaling for Solr on Kubernetes.
SolrCloud clusters are often large and complex, with each organization using its own code to deploy and maintain these clusters. The Solr Operator was a first step in consolidating complexity through official deployment tooling to run Solr on Kubernetes. However the Operator does not address scaling up and down based on demand.
Much like it provides generic deployments, Kubernetes provides generic ways of autoscaling applications, such as the HorizontalPodAutoscaler (HPA). This works especially well for stateless applications, much like deployments do. Solr is a stateful application that has specific state assigned to each pod (Solr node), therefore autoscaling SolrClouds with the HPA will not work by default.
The Solr Operator has already been built to extend Kubernetes’ StatefulSets, Services and Ingresses to support Solr’s unique use-case. Therefore it is the prefect mechanism to also bridge the gap between the HorizontalPodAutoscaler and Solr.
Through extending the functionality of the Solr Operator, and adding new APIs to Solr, we will show how autoscaling can be re-introduced to the Solr ecosystem.
Houston is a Lucene/Solr PMC member and committer. He works at Apple on the Open Source Technologies team, developing Solr and creating a better ecosystem for it in the cloud. Previously Houston worked at Bloomberg, as a member of the Search Infrastructure team. He has degrees in Computer Science & Mathematics from The University of Texas at Austin.