I have Dask Gateway deployed to my kubernetes environment and am able to specify a custom image for the deployment. The basic example of:
a = da.random.normal(size=(1000, 1000), chunks=(500, 500))
a.mean().compute()
Works. What I would like to do is allow the user to create a cluster with different images/tags from the one configured in the Helm chart for Dask Gateway. It’s not clear to me how I would do that.
Even more basic examples, taken from the Dask Gateway documentation don’t seem to work. For example:
cluster = gateway.new_cluster(worker_cores=2, environment="tensorflow")
Gives:
ValueError: Unknown fields ['environment', 'worker_cores']
With dask_gateway.__version__ reporting 2023.9.0
Hi @dcieslak, welcome to Dask discourse!
In order to implement such functionnality, you want to look at how to define the options user can configure. This is a little further in the documentation:
https://gateway.dask.org/cluster-options.html#server-configuration
You could perfectly define a list of image to select from, or even a plain String field.
1 Like
Follow up question - I was able to use the technique above to set things like image and worker_cores, basically values that I see in Configuration Reference — Dask Gateway 2023.9.0 documentation
Now, lets say that I have 2 nodepools (to use an AKS term), one with GPU and another without. Nodes in each pool have different taints and labels.
How would a user be able to request a Dask cluster with nodes with GPUs (more generally tolerating a certain taint and setting a label) for one workload and a different Dask cluster (more generally tolerating a different taint (or no taint) and setting a different label)?
Much appreciated!
According to the KubeClusterConfig documentation, you should be able to define things like worker_extra_pod_annotations, worker_extra_pod_labels and others using the Option mechanism.