Does anyone use airflow and dask at once？
Hi @namnam and welcome! In this blog post, the first use case describes a setting where Dask and Airflow (or Prefect) can be used together, alongside Kubernetes (or another resource manager). Would you be able to provide a bit more detail on what you’d like to do?
Very quick overview: you can launch Dask clusters from within Airflow tasks and run heavy computations on that cluster. Because of the way Airflow’s XCOM system is designed, it’s not possible to transfer results stored in cluster memory across Airflow tasks. If you want to run multiple tasks on the same Dask cluster I recommend looking into using the Airflow DaskExecutor.