2024-06-10 –, Maschinenhaus
We’re taking a stab at a definitive - yet nuanced! - answer to “how much heap do I need?” and “is GC a problem and how can I fix it?” for Solr/Elasticsearch/OpenSearch deployments.
If the Java heap is too low, you may hit an OOM or circuit breakers. Or maybe the GC latency spikes, causing cluster instability. GC latency can also be high when cleaning a heap that’s too large. But even if it doesn’t, you might be wasting memory that can be used by the OS to cache index files.
Choosing good heap size and GC options (i.e. for your use-case) is important for both stability and performance. In this session, we’ll start by looking at what uses heap (we’ve got some cool production stories in this area) and how garbage is being generated and cleaned up. Then, we’ll zoom into what happens when you run out of heap: you either simply need more, or your GC fell behind. We’ll look at how you can know whether you need more and which GC options allow you to improve throughput. Sometimes at the expense of median latency, but usually p99 or maximum latency is what we really care about.
The aim is for participants to understand how to use metrics and knowledge about the use-case to choose the heap size and GC options. Instead of using rules of thumb like “use half of your RAM but no more than 30GB heap” or ParallelGCThreads=4. There’s a grain of truth in those, but they can be very limiting on large-scale clusters (and in some small clusters, too). We’ll cover both the origins and the limitations of these rules of thumb as well.
Radu Gheorghe works mainly as a search consultant at Sematext, working with clients of all sizes on their Elasticsearch, OpenSearch and Solr projects. He is also a trainer and does production support for both these search engines.
Sometimes he helps out with the development of Sematext Cloud (an observability SaaS), mostly when it comes to Elasticsearch and log shippers (e.g. Logstash, rsyslog…). He also writes on the Sematext blog or helps other publish new articles.
He co-authored a book (Elasticsearch in Action, Manning), recorded a video tutorial (Working with Elasticsearch, O'Reilly) and was a speaker at a number of conferences, such as Berlin Buzzwords, LuceneSolrRevolution (later Activate) and Kubecon.
Software engineer, trainer, consultant and author from time to time - some would say that he is an all in one battle weapon concentrated mostly on Lucene, Solr and Elasticsearch. Currently an Engineering Lead in Archipelo. However he also likes all the other cool stuff that is happening in the IT world. Likes to share his knowledge by giving talks at various meet ups and conferences.