Multi-tenant in Airflow is almost there

Photo of Oksana Lyniv by Oliver Wolf pic (License CC BY 4.0) When considering multi-tenancy, it is often associated with SaaS applications. However, most of my clients do not seem to be involved because they typically address internal needs. But these days, a new paradigm is rising. It's called Platform Engineering. And it doesn't target only operational software but also data related products. We need to develop accessible data tools for corporate end users, sush as dashboards and ETL services. Therefore, providing Airflow with multi-tenant capabilities is essential. Nearly four years ago, someone questioned the community on Stack Overflow about how to provide a multi-team feature with Airflow. Jarek Potiuk, a main Airflow contributor, provided a very comprehensive answer. In summary, he explained that Airflow did not yey provide a multi-tenancy feature. He went further adding that, even if in the next months this could be designed and implemented in Airflow 3, from it side, the multi-tenancy should be still adressed with multiple Airflow instances in some context where isolation is a must have. Finally, AIP-1, an Airflow Improvement Proposal, aimed to enhance the Airflow security in all the dag lifecycle (from submission to execution), starts feeding the discussion of the multi-tenancy design. The AIP-43 and AIP-44 provided an initial solution to implement this feature as early as Airflow 2.

Mar 17, 2025 - 10:10

Photo of Oksana Lyniv by Oliver Wolf pic (License CC BY 4.0)

When considering multi-tenancy, it is often associated with SaaS applications. However, most of my clients do not seem to be involved because they typically address internal needs.

But these days, a new paradigm is rising. It's called Platform Engineering. And it doesn't target only operational software but also data related products.
We need to develop accessible data tools for corporate end users, sush as dashboards and ETL services.

Therefore, providing Airflow with multi-tenant capabilities is essential.

Nearly four years ago, someone questioned the community on Stack Overflow about how to provide a multi-team feature with Airflow. Jarek Potiuk, a main Airflow contributor, provided a very comprehensive answer. In summary, he explained that Airflow did not yey provide a multi-tenancy feature. He went further adding that, even if in the next months this could be designed and implemented in Airflow 3, from it side, the multi-tenancy should be still adressed with multiple Airflow instances in some context where isolation is a must have.

Finally, AIP-1, an Airflow Improvement Proposal, aimed to enhance the Airflow security in all the dag lifecycle (from submission to execution), starts feeding the discussion of the multi-tenancy design.

The AIP-43 and AIP-44 provided an initial solution to implement this feature as early as Airflow 2.