Different machine types for scheduler & workers

Distributed
1

Hello.
How can I run a scheduler in GCP that isn’t the same machine type as workers (fewer than workers)?
Thanks.

2

Thanks for the question @tarasenkoworld. Looking at the dask_cloudprovider.gcp.GCPCluster docstring, it looks like there’s only a single machine_type option which applies to the scheduler and all workers. This isn’t a limitation in Dask itself (e.g. the dask_cloudprovider.aws.EC2Cluster cluster manager supports scheduler_instance_type and worker_instance_type options), I suspect it’s just that nobody has gotten around to implementing them for GCPCluster yet. I’ve gone ahead and opened this issue Support different machine types for scheduler and workers on GCP · Issue #390 · dask/dask-cloudprovider · GitHub asking for this to be added to GCPCluster.

FWIW Coiled does support specifying different instance types on GCP. @jacobtomlinson are there other cluster managers / workaround you’re aware of for this?

1 Like