Installing Dask Workers on Partcular Node Pool

Distributed
,
1

Hi Team,

We would like to use Dask Hub with Gateway Cluster and have 2 Nodepools (one for CPU and one for GPU) and we want to deploy the Dask Workers on GPU nodes , we have tried different configuration to set the node selector and taint but nothing worked and dask workers are creating in CPU pool only , Please find the config file for your reference

backend:
      node_selector: {'app.kubernetes.io/name': 'gpupoc'}
      tolerations: [
        {
          'key': 'sku',
          'operator': 'Equal',
          'value': 'gpu',
          'effect': 'NoSchedule'
        }
      ]
      worker:
        extraPodConfig:
          node_selector: {'app.kubernetes.io/name': 'gpupoc'}
          tolerations: 
            - 'key': 'sku'
              'operator': 'Equal'
              'value': 'gpu'
              'effect': 'NoSchedule'
        node_selector: {'app.kubernetes.io/name': 'gpupoc'}
        tolerations: [
          {
            'key': 'sku',
            'operator': 'Equal',
            'value': 'gpu',
            'effect': 'NoSchedule'
          }
        ]
      cores:
          request: 2
        memory:
          request: "4 G"
        threads: 2

Code which we are using to create dask cluster

from dask_gateway import Gateway
gateway = Gateway()
clusters = gateway.list_clusters()
cluster = gateway.new_cluster(shutdown_on_close=False)
cluster.adapt(minimum=2, maximum=4)
client = cluster.get_client();

Can you please check and guide how we can select node selectors and taints for dask worker nodes and how to set no of threads in dask worker nodes as nthread configuration in helm config file not working .

2

@akash-goel Thanks for your question! I’m not familiar with Dask+GPUs, but @jacobtomlinson might have thoughts on this. :smile:

3

I think this might be the same issue:

I would like to deploy my worker nodes to a separate GKE node pool from the scheduler/controller. Is that possible? It doesn’t seem I can set the nodeSelector property for worker pods?

4

Hi @ms0, welcome to Dask comunity!

I think the only way is to configure the extraPodConfig fields in your yaml file.

I’m no Kubernetes expert, but there is an example in the link above, and you should be able to configure nodeSelector property with it. The example from the first pod of this topic doesn’t seem right when looking at the example or at Kubernetes documentation.