Custom cluster-spec.ymal file does not create workers: KubeCluster

Distributed
1

Hi all,
I am trying to setup Kubecluster using custom cluster-spec.ymal, however; it shows number of workers as zero.

I am trying on a local machine with WSL.

Here are the spec:

# cluster-spec.yml
apiVersion: kubernetes.dask.org/v1
kind: DaskCluster
metadata:
  name: example
  labels:
    foo: bar
spec:
  worker:
    replicas: 2
    spec:
      restartPolicy: Never
      containers:
      - name: worker
        image: "ghcr.io/dask/dask:latest"
        imagePullPolicy: "IfNotPresent"
        args: [dask-worker, --nthreads, '2', --no-dashboard, --memory-limit, 1GB, --death-timeout, '60', '--name', $(DASK_WORKER_NAME)]
        env:
          - name: EXTRA_PIP_PACKAGES
            value: git+https://github.com/dask/distributed
        resources:
          limits:
            cpu: "1"
            memory: 1G
          requests:
            cpu: "1"
            memory: 1G
  scheduler:
    spec:
      containers:
      - name: scheduler
        image: "ghcr.io/dask/dask:latest"
        imagePullPolicy: "IfNotPresent"
        args:
          - dask-scheduler
        ports:
          - name: tcp-comm
            containerPort: 8786
            protocol: TCP
          - name: http-dashboard
            containerPort: 8787
            protocol: TCP
        readinessProbe:
          httpGet:
            port: http-dashboard
            path: /health
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          httpGet:
            port: http-dashboard
            path: /health
          initialDelaySeconds: 15
          periodSeconds: 20
    service:
      type: ClusterIP
      selector:
        dask.org/cluster-name: example
        dask.org/component: scheduler
      ports:
      - name: tcp-comm
        protocol: TCP
        port: 8786
        targetPort: "tcp-comm"
      - name: http-dashboard
        protocol: TCP
        port: 8787
        targetPort: "http-dashboard"

8
81377×622 38.7 KB

Logs:
9
91586×705 79.7 KB

At the same time , I was able to work with python code, that is :

10
101247×291 24.5 KB

Can someone help me to identify and solve the issue with .ymal file?

Thanks in an advance

2

Hi @Nirajkanth,

So just to be sure, KubeCluster works when you use kwargs other than custom_cluster_spec, right?

So you think there is an issue in your yaml file somewhere.

Are you able to issue kubectl commands to see the status of your KubeCluster and associated pods? It would be worth looking at the Worker pods, are they started? What are the output logs?