2025-06-16 –, Maschinenhaus
If you were living under the rock and have not heard that Airlfow 3 is out, and solves most of the pain points that you had with Airflow 2, this talk is for you. You will learn how you can boost your Data Engineering and AI/ML workflows (without having to rewrite your DAGs) with what Airlfow 3 community worked for last 12 months.
Airflow 3 is out. Spring last year Airlfow community came to conclusion that in order to respond to a number of users of ours, we have to again reinvent ourselves and release Airflow 3. Berlin Buzzwords is at the right time and the right place to talk about it, as at the time of Buzzwords, Airflow 3 will already be out for a while and we will know not only what we planned, but how our users already use Airflow 3 and what benefit it brings for them.
In this talk you will learn from Jarek, one of the top maintainers of Airflow all about you want to know about Airflow 3:
- why we decided to change to Airflow 3
- what are the architectural changes and improvements that lay foundation under Airflow 3 being modern and applicable to more workflows
- what are the features you always wanted and you can use now: Versioning, Enterprise level security isolation, better dependency management, execution isolation, dataset as first class citizen, modern react-based UI, schedulable and UI-controlled backfills, ML/AI worfklows including inference workflows, almost streaming experience, and more
- what early users of Airlfow 3 say and how their workflow management improved
- what is coming next (yes! we are not nearly done yet and more things are coming!)
Last year, I presented "new orchestrator in town" - Airflow 2, but that was only a Prelude. If you want to hear the whole Symphony come to see the talk.
Data Science, Stream, Operations
Level:Intermediate
Independent Open-Source Contributor and Advisor, Committer and PMC member of Apache Airflow, Member of the Apache Software Foundation, Security Committee Member of the Apache Software Foundation. Organizer of community-focused events, speaker.
Jarek is an Engineer with a broad experience in many subjects - Open-Source, Cloud, Mobile, Robotics, AI, Backend, Developer Experience, Security, but he also had a lot of non-engineering experience - building a Software House from scratch, being CTO, organizing big, international community events, technical sales support, pr and marketing advisory but also looking at legal aspects of security, licensing, branding and building open-source communities are all under his belt.
With the experience in very small and very big companies and everything in-between, Jarek found his place in Open-Source world, where his internal individual-contributor drive can be used to the uttermost of the potential.