Berlin Buzzwords 2024

Sequential Testing Simplified with Basemath
2024-06-11 , Frannz Salon

Learn how sequential testing has cut our experiment durations by nearly 40%. We'll delve into the core principles, tackle real-world implementation issues, and unveil Basemath, our Python-driven open-source tool. Gain insights on refining your experimental analysis for smarter feature deployment.


At GetYourGuide, we've achieved a 40% reduction in experiment timelines by adopting sequential testing, a method that aligns intuitive decision-making with statistical accuracy. This approach allows for dynamic stopping rules based on real-time data, ensuring the validity of statistical conclusions. Sequential testing is increasingly popular in the tech industry for its ability to streamline the traditional A/B testing process, which typically requires a fixed sample size and can lead to unnecessary delays when experimenters have an early indication of an experiment's outcome.

In our upcoming presentation, we will delve into the foundational concepts of sequential testing and share practical experiences from our own implementation, including the challenges we faced. We'll introduce Basemath, our own approach to sequential testing that eliminates the need for repeated numerical integration, enhancing efficiency and accessibility.

Basemath's Python-based implementation speeds up the evaluation of multiple concurrent experiments and is more user-friendly than traditional C++ code. As an open-source project, Basemath is available to the broader community, offering a valuable tool for anyone seeking to optimize their experimental analysis.

Join us to learn how sequential testing can transform your A/B testing strategy and help you make more informed decisions about feature rollouts.

See also: [Preprint] Basemath’s Test: Group Sequential Testing Without Recurrent Numerical Integration (351.8 KB)

Dr. Alexander Weiss is a Senior Data Science Manager at GetYourGuide, overseeing the Data Product teams within the Marketplace division. Holding a Ph.D. in Probability Theory from TU Berlin, he brings more than ten years of expertise in data science and analytics. While his teams primarily focus on developing data products in the realms of search and relevance, Alexander harbors a deep passion for experimentation statistics. He is passionate about empowering organizations to make data-driven decisions and is excited to share his insights on sequential testing at the upcoming conference. His hobbies are on hold until the children have moved out.

Konrad Richter is a Senior Data Analyst at GetYourGuide, developing their internal experimentation platform and enabling product teams to make data driven decision. Previously he worked on pricing-related topics at Zalando and Mercedes-Benz, and holds a masters' degree in Computational Data Analytics from the Georgia Institute of Technology.