Airflow and Dask

Distributed
1

Does anyone use airflow and dask at once？ :smiling_face_with_tear:

2

Hi @namnam and welcome! In this blog post, the first use case describes a setting where Dask and Airflow (or Prefect) can be used together, alongside Kubernetes (or another resource manager). Would you be able to provide a bit more detail on what you’d like to do?

3

Hi @namnam - there are also some specific Airflow DAG examples that use Dask in this tutorial post.

Very quick overview: you can launch Dask clusters from within Airflow tasks and run heavy computations on that cluster. Because of the way Airflow’s XCOM system is designed, it’s not possible to transfer results stored in cluster memory across Airflow tasks. If you want to run multiple tasks on the same Dask cluster I recommend looking into using the Airflow DaskExecutor.

4

Thanks @rrpelgrim @scharlottej13
I configured airflow according to the example. When I did not start the dask service, the task was executed successfully！In the jobs page, the executor class displays DaskExecutor. :rofl: