A core part of our business is to receive and then process humongous amounts of financial data from all over the globe. This pipeline scales to tens of billions of pricing messages every day, in which each message carries highly valuable information. Getting visibility into what was sent to us versus what was published to our customers is of utmost importance to enable internal teams to quickly troubleshoot issues reported by any of the data providers.

But how do we capture essential information about the data flowing through such a massive and high throughput system scaling to more than tens of thousands of processes running on close to a hundred machines, in which traffic peaks at more than a million messages per minute? In this talk, we will talk about how we built a high throughput telemetry system for streaming, storing, and searching such a high volume of data, starting from scratch using open source technologies like ZeroMQ, Apache Kafka, Kubernetes, and Apache Cassandra. You will gain valuable insights into the system’s design and performance, as well as the lessons we learnt along the way. We will cover everything from schema design and load testing to incremental deployment in order to manage such high data throughput.