2024-06-11 –, Maschinenhaus
You've heard the wisdom of choosing boring technology, but how do you balance this with new usage patterns and ever-increasing scale? Learn how Slack developed a petabyte-scale log search engine with a team of just three engineers, supporting over one million queries per day.
In early 2021, and after yet another incident caused by a large spike in log volume, it was clear that something had to change. Facing a near-constant 50% year-over-year volume growth and rapidly mounting operational overhead, the Slack Observability team made the decision to build a new open-source log search engine.
Building a new log search engine is a large undertaking, and there are many features that engineering teams require before replacing an existing solution. Many of these are solved problems however, with projects like Apache Lucene and Opensearch. At Slack our focus became how we could build on these existing, boring technologies to solve our unique scaling challenges better. This enabled us to support features specific to log search, and ease our team’s operational challenges by building a solution that was tailored for Slack infrastructure.
In this talk we will discuss how to identify areas where using perceived boring technology can reduce risk and accelerate development. Learn how to do more with fewer people, and how to see further, standing on the shoulders of giants.
Bryan Burkholder is a Staff Software Engineer at Slack, focused on improving observability adoption across the engineering organization. Recently this work has centered around developing an open-source log search and analytics engine that can handle petabyte scale in a cost-effective manner.
Varun Thacker is a Staff Software Engineer at Slack, currently focused on log search for observability data. He is an Apache Lucene and Solr committer and Project Management Committee member. Previously, he has worked on search at Slack and Lucidworks.