How does the dask config works behind the scenes?

Distributed
,
1

Hi
im trying to learn as deep as possible the dask config and based on docs im not so confident i 100% correctly understand it.

what is the order of execution for when its applied, how does it affect the the logic, scheduler, graph etc?

in k8s it does not modify the config at run time and its possible to modify via the env vars them self by directly sending it to the workers before they are initialized.

that being said - im trying to understand how does it affect the scheduler , graph etc.?

based on this
https://docs.dask.org/en/stable/phases-of-computation.html

i understand that the graph is built by the scheduler and not the workers so maybe modifying on run time the scheduler does work?

any help would be appreciated thx!

2

Hi,

There are several things mentionned in the docs. Notably:

  • an example showing the modifying configuration from code can affect the graph construction an other things.

  • Finally, note that persistent objects may acquire configuration settings when they are initialized. These settings may also be cached for performance reasons. This is particularly true for dask.distributed objects such as Client, Scheduler, Worker, and Nanny.

So when using Distributed, other started process will not be affected by the local config object.

1 Like