2024-06-10 –, Frannz Salon
What's the role of Orchestrators in the Modern Data Stack?. Trying new orchestrators? Do we need Orchestrators at all? Or what "Orchestrator" can do for you had been redefined ?
Meet "new" orchestrator in town ..... "Apache Airlfow 2.x" with ways of orchestration you had not realized you can do.
Efficient and well managed Orchestrating your Data processing Pipelines is crucial to provide the back-bone of the modern Data processing needs. With all the new rage of LLMS and AI, it's more important than ever to make sure that you can wrap your head around all the old, new and upcoming data processing tools and services you use, but also track the provenance, lineage of the data, and make it easy to author and mange those for distributed teams.
This talk will bring to the light how modern Airflow 2.8+ provides proven and also modernised, easy and nice to use ways on how to do it. Quite often you hear about the "new" orchestrator that aims to solve your orchestration needs. You can also often hear how it compares to Airflow.
However those comparisions often overlook the fact that since Airflow 2.0 has been introduced, it continues to evolve and piece-by-piece modernize itself and respond to the needs of processing even more data, and interacting with even more systems that were not even existing a year ago (think LLMs) - while continue harnessing the powers of other tools you already used in the past and bind them all together.
New UI, New ways of writing your orchestration tasks, new ways to test them, tracking lineage, simpler authoring and intracting with object storages. And the comparision often overlook that if you start your journey with Airflow today, your experience will be quite a bit different than even 2 years ago (what usually most comparisions talk about). This talk will highlight some of the important ones.
Also you might be surprised but this is all happening without breaking compatibility - hence you can still you use it in the "old way".
But maybe it's time to learn new ways?
Independent Open-Source Contributor and Advisor, Committer and PMC member of Apache Airflow, Member of the Apache Software Foundation, Security Committee Member of the Apache Software Foundation
Jarek is an Engineer with a broad experience in many subjects - Open-Source, Cloud, Mobile, Robotics, AI, Backend, Developer Experience, Security, but he also had a lot of non-engineering experience - building a Software House from scratch, being CTO, organizing big, international community events, technical sales support, pr and marketing advisory but also looking at legal aspects of security, licensing, branding and building open-source communities are all under his belt.
With the experience in very small and very big companies and everything in-between, Jarek found his place in Open-Source world, where his internal individual-contributor drive can be used to the uttermost of the potential.