Dask-Kubernetes not cleaning up properly

Hi,

I’m creating ephemeral Dask Clusters on K8S (AWS EKS) to run some ETL jobs.
Unfortunately, the “dask-root” pods are never stopped and deleted.

Is there a config flag I can set for a full cleanup?

At the moment, my cluster accumulates a one idling dask-root pod per finished ETL run:

Code:

# see https://kubernetes.dask.org/en/latest/
def dask_pod_spec() -> V1Pod:
    return make_pod_spec(
        image="ghcr.io/dask/dask:latest",
        memory_limit="1G",
        memory_request="1G",
        cpu_limit=1,
        cpu_request=1,
        extra_pod_config={"serviceAccountName": "prefect-agent-service-account"},
    )


# https://kubernetes.dask.org/en/latest/kubecluster.html
dask_runner = DaskTaskRunner(
        cluster_class=dask_kubernetes.KubeCluster,
        adapt_kwargs={"minimum": 1, "maximum": 10},
        cluster_kwargs={
            "pod_template": dask_pod_spec(),
            "env": {
                "EXTRA_PIP_PACKAGES": "prefect==2.0b11"
            }
        },
    )

@rahloff Welcome to Discourse!

I’m not familiar with this, but maybe @guillaumeeb or @jacobtomlinson have some thoughts. :smile:

1 Like

Hi pavithraes, thanks for making the introductions!
@guillaumeeb / @jacobtomlinson, any idea or pointers where I might need to look?