Berlin Buzzwords 2024

Fixin the Hard Bits of Event Processing with Restate & Kafka
2024-06-10 , Kesselhaus

Discover Restate: A new tool for event-driven apps, blending Kafka's power with simplicity for handling complex transactions, async tasks, and microservices. Dive into its unique approach to tackle scalability, resilience, debuggability, and state management challenges in our session.


Event-driven applications are still hard in 2024, despite all their strong and desirable properties (scalability, resilience, ability to replay, ...). While streaming analytics are supported by a wave of powerful tools (like Flink, KStreams, Materialize, RisingWave, ...), the transactional side of event-driven applications is not quite as far.

Complex per-event control flow, complicated state semantics, and interactions with many external dependencies (APIs/services/...) make those applications challenging. Observability and debuggability are still ways behind what RPC-based architectures offer with today's tools.

In this talk we present 'Restate' (https://restate.dev/), a new open source tool we have been working on. Restate complements systems like Kafka when building event-driven workflows, async tasks, distributed signaling, or even full microservices architectures. While Restate internally is an event-driven system (sharing some similarities with a Kafka/Flink/KStreams crossover), it has a very simple programming model: Durable Execution (for event-handlers and RPC), Durable Promises (for signaling), and Virtual Objects. Those three constructs are the resilient encapsulations of the core ingredients of every distributed application: compute, communication, and state.

In the talk, we will first we'll look practically at common challenging patterns: Where do those tend to break current architectures? How can we address those issues with Kafka/Restate?
- burst-y loads / rapid scaling
- head-of-the-line waiting on long event processing times
- combining Kafka events with webhooks and other signals
- Transactionally updating databases
- high-fan-out routing to consumer APIs
- state machines with long-running transitions
- delaying/postponing certain events

Second, we take a look at similarities, synergies, and differences between Restate, Kafka, and Stream-Processing. We look at the way applications are expressed, how the systems organize event logs and event processing, and how this mirrors the differences between analytical- and transactional applications respectively.

See also: Slides (1.5 MB)

Stephan is one of the original creators and of Apache Flink and founder / CTO of dataArtisans/Ververica. Recently, he co-founded Restate (https://restate.dev/) with the goal to simplify distributed application development, event-driven app, and microservice architectures.