Setting Dask Distributed config variables when deploying Dask Gateway with Helm

Deploying Dask
,
1

Hi all, I’d like to set some of the parameters as described in the Dask Distributed docs such as worker spill and allow-failures but don’t know where to set these variables when using the Dask Gateway helm charts

I have tried using dask.config.set from the calling code but I found that the configuration does not carry over. I also haven’t had any success trying to set the configuration using env vars such as DASK_DISTRIBUTED__SCHEDULER__ALLOWED_FAILURES

2

@LewisJarrod Hi, and welcome to Discourse! Have you tried setting these using the YAML config files?

I have tried using dask.config.set from the calling code but I found that the configuration does not carry over.

Some config values are read only once, when the scheduler/workers are created. My guess is something like that might be happening here…

I also haven’t had any success trying to set the configuration using env vars such as DASK_DISTRIBUTED__SCHEDULER__ALLOWED_FAILURES

This is strange. To confirm, are you setting these variables at the very being of your workflow? (Same as above, if they’re set later, they may not be read)

If the yaml config also doesn’t work, I’d encourage you to open an issue on the dask-gateway issue tracker to reach the developers directly and to confirm that it’s not a bug. :smile:

3

Thanks for the respnse! I didn’t get a notification in my email so it took me a bit to circle back here. In this case since I’m using a Dask Gateway, I don’t actually see the distributed.yaml file, I assume a version of it exists behind the scenes somewhere.

I have tried to set dask.config.set before my cluster is created and have set the environment variables on the scheduler + worker pods, and have confirmed they are being set, but the deployment doesn’t seem to do anything with the information.

I added an issue to the dask-gateway github as well

2 Likes
4

We found the answer to this, if we write a config.yml then copy it to the expected location (~/.config/dask/config.yml) in our Dockerfile that we use for the workers then the values are picked up correctly in the deployment

1 Like