Berlin Buzzwords 2024

How we isolate streaming ingest from search using RocksDB
2024-06-10 , Maschinenhaus

Learn a new cloud architecture that separates streaming ingest compute from query compute for real-time applications and how it addresses the age-old issue of compute contention.


In this presentation, we delve into how we disaggregated streaming ingest compute from query compute for real-time applications. Up until this point, real-time architectures were designed for data streaming ingestion and query processing to be executed within the same cluster. This co-location, while beneficial for fresh data access, often leads to competition for compute resources, especially during spikes in either data ingestion or query activities. There are strategies for mitigating compute contention, including replication and overprovisioning clusters, but they do not fully address the issue.

We describe how we built a new cloud architecture using RocksDB, a key-value store with a log-structured merge-tree architecture. The design disaggregates compute from storage, enabling simultaneous querying of shared real-time data across multiple clusters. The architecture also separates streaming ingest compute and query compute by replicating the in-memory state of the RocksDB memtable across different compute clusters, ensuring the accessibility of the latest data in single-digit milliseconds. Learn how this architecture is being used for search, real-time analytics and AI-powered search applications.

See also: Slides (1.5 MB)

Igor Canadi is a founding engineer and architect at Rockset working on the distributed SQL query engine and cloud-native architecture. Previously, Igor was an engineer at Facebook, working on the database engineering and product infrastructure teams, where he contributed to RocksDB, developed MongoRocks and MongoDB with RocksDB storage engine, drove RocksDB open source initiatives, worked on core GraphQL infrastructure for Facebook’s Android application, and owned GraphQL developer tooling for hundreds of developers. Igor holds a master’s degree in computer science from the University of Wisconsin-Madison and a bachelor’s degree from the University of Zagreb. In his free time, he likes sailing and snowboarding.