Using Dask Gateway to create clusters with different images

I have Dask Gateway deployed to my kubernetes environment and am able to specify a custom image for the deployment. The basic example of:

a = da.random.normal(size=(1000, 1000), chunks=(500, 500))

Works. What I would like to do is allow the user to create a cluster with different images/tags from the one configured in the Helm chart for Dask Gateway. It’s not clear to me how I would do that.

Even more basic examples, taken from the Dask Gateway documentation don’t seem to work. For example:

cluster = gateway.new_cluster(worker_cores=2, environment="tensorflow")


ValueError: Unknown fields ['environment', 'worker_cores']

With dask_gateway.__version__ reporting 2023.9.0

Hi @dcieslak, welcome to Dask discourse!

In order to implement such functionnality, you want to look at how to define the options user can configure. This is a little further in the documentation:

You could perfectly define a list of image to select from, or even a plain String field.

1 Like

Follow up question - I was able to use the technique above to set things like image and worker_cores, basically values that I see in Configuration Reference — Dask Gateway 2023.9.0 documentation

Now, lets say that I have 2 nodepools (to use an AKS term), one with GPU and another without. Nodes in each pool have different taints and labels.

How would a user be able to request a Dask cluster with nodes with GPUs (more generally tolerating a certain taint and setting a label) for one workload and a different Dask cluster (more generally tolerating a different taint (or no taint) and setting a different label)?

Much appreciated!

According to the KubeClusterConfig documentation, you should be able to define things like worker_extra_pod_annotations, worker_extra_pod_labels and others using the Option mechanism.