2023-06-19 –, Frannz Salon
How we successfully transitioned the search system for the world's largest recipe sharing platform to a modern stack – including the successes, failures, team structures, and processes along the way.
Cookpad is the largest recipe sharing platform in the world. Our mission is to make everyday cooking fun, and central to that is our search product. Our search engine helps cooks everywhere find tasty dishes to cook from our ever-growing catalogue of six million recipes created by everyday cooks. As a global recipe search – available in 70+ countries, 30+ languages, and to over 50 million monthly users – delivering this is no mean feat.
In order to prepare for a substantial new iteration of our search product, we realised that our existing search system was not suited to our goals. Over the course of two years, we embarked on a technological and cultural transition, with the aim of giving product teams and engineers (including search engineers, data scientists, and ML engineers) greater ownership over the search experience. This included a shift to a Python-based stack and a data-driven approach to search relevance improvement.
We embarked on a transition to a new system, along with new team structures and team composition, and new strategies for improving the search experience. Over two years we delivered a new system, without halting product development along the way, and without disruption to the user experience.
Our starting point was a team and system with capacity limited to maintenance and bug fixes, where relevance enhancement was delivered through incremental knowledge base tuning by SMEs (non-engineer subject matter experts). Our end point was multiple search teams who have greater ownership over the search experience and relevance improvement, assisted by SMEs, and following a process for rigorously tested hypothesis-driven experimentation.
This change involved transitioning to a new event-driven architecture, along with technologies that were new to Cookpad search, such as Kubernetes, Kafka, Python, and machine learning. In addition – and just as importantly – it also involved a transformation in team structures and team composition, for which we borrowed many concepts and practices learned from the search community, and also ideas from the Team Topologies movement.
This talk will cover our journey, why we did it, as well as the trials, tribulations, and successes along the way. Hopefully, it will give others who are in a similar position new ideas on how to reinvent their own search system and search function, while minimising disruption to product delivery, in order to deliver proven improvements at pace.
Matt Williams is a Principal Engineer at Cookpad, the world's largest recipe sharing platform, where he specialises in search and discovery. He has over six years of experience in building discovery experiences, with a particular interest in NLP, ML, scaleable search and recommendation, and the team structures and processes that underpin effective relevance improvement.
Prior to joining Cookpad in 2019, Matt worked as a Data Scientist and ML Engineer in industry, with applications to social media analysis and real-time news analysis. Before entering the software industry, Matt was a research scientist in academia, focusing on network science and predictive user modelling. He holds a PhD in Computer Science from Cardiff University, where he was also a lecturer.