# Dask config - how does it actually work?
**URL:** https://dask.discourse.group/t/dask-config-how-does-it-actually-work/3097
**Category:** Distributed
**Tags:** kubernetes, distributed
**Created:** [September 2, 2024, 9:00am UTC](https://dask.discourse.group/t/dask-config-how-does-it-actually-work/3097 "2024-09-02T09:00:37Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Hvuj](https://dask.discourse.group/u/Hvuj)
#### Post date: [September 2, 2024, 9:00am UTC](https://dask.discourse.group/t/dask-config-how-does-it-actually-work/3097/1 "2024-09-02T09:00:37Z")
Hi.
Im trying to learn what is the correct way to use dask.config.
im using distributed environment using k8s on gcp (vms)
scenario:
i have input function logic function and output function.
input - in charge of reading parquet files with different settings
logic - the actual transformation that happens
output - in charge of writing to parquet with different configurations.
i would like to start to use a custom dask.config using env vars , yaml and context manager.
im trying to understand how does it work behind the scenes.
for example:
in the input function i will have a context manager that has read parquet e.g.
```auto
with dask.config(some logic):
return dd.read_parquet(...)
```
(no compute nor persist)
the ddf is then passed to the logic function which also has with `dask.config(some logic):`
it could have some times persist, compute , set\_index (not lazy one) etc. and some times pure lazy logic.
it goes to the output function which also has
```auto
dask.config(some logic):
dd.to_parquet(some logic)
```
i do understand that the config tells how the graph is going to be built e.g. what configuration it should use , but what i dont understand which config and when will it use
thanks for the help
---
### Author:  [@jacobtomlinson](https://dask.discourse.group/u/jacobtomlinson)
#### Post date: [September 2, 2024, 9:18am UTC](https://dask.discourse.group/t/dask-config-how-does-it-actually-work/3097/2 "2024-09-02T09:18:20Z")
It would help to understand what you’re trying to do here.
---
### Author:  [@Hvuj](https://dask.discourse.group/u/Hvuj)
#### Post date: [September 2, 2024, 10:45am UTC](https://dask.discourse.group/t/dask-config-how-does-it-actually-work/3097/3 "2024-09-02T10:45:19Z")
im trying to optimize performance depending on the logic in the transformation and squeeze performance when possible.
for example:
if the logic is quite heavy im trying to spill to disk more often - play with worker saturation
(example lets say i use below but it could be any logic here)
```auto
"distributed.worker.memory.spill": 0.1,
"distributed.worker.memory.target": 0.8,
"distributed.worker.memory.pause": 0.9,
"distributed.worker.memory.terminate": 0.95,
"distributed.worker.memory.spill-compression": "auto",
"distributed.scheduler.work-stealing": True,
"distributed.scheduler.worker-saturation": 1.5,
"dataframe.shuffle.method": "p2p",
"dataframe.shuffle.compression": "auto",
"distributed.nanny.environ.MALLOC_TRIM_THRESHOLD_": "1024",
```
or could be even on the array or dataframe level configurations and etc…
so to give some more context - i can pass from any logic function dynamically dask config dict to the output function.
so what i want is:
1. have some extended defaults for my k8s cluster (for that i use .yaml)
2. input function should have some additional logic if we want and if not it will use dask.config.config defaults
3. output function should have some extended defaults and if necessary use dynamically what we pass from the logic function.
the thing is im not 100% under what conditions which config will be used - depending on the answer i will change the implementation
---
### Author:  [@jacobtomlinson](https://dask.discourse.group/u/jacobtomlinson)
#### Post date: [September 2, 2024, 1:25pm UTC](https://dask.discourse.group/t/dask-config-how-does-it-actually-work/3097/4 "2024-09-02T13:25:39Z")
Unfortunately Dask’s config system is not as granular as this. These kinds of config options are applied when the workers start up and can’t be modified at runtime.
If you’re using `dask-kubernetes` you can dynamically create Dask clusters with `KubeCluster` and apply these configs to each cluster.
---
### Author:  [@Hvuj](https://dask.discourse.group/u/Hvuj)
#### Post date: [September 2, 2024, 1:26pm UTC](https://dask.discourse.group/t/dask-config-how-does-it-actually-work/3097/5 "2024-09-02T13:26:47Z")
thx - so to give more context every function has an ephemeral k8s cluster- so essentially im applying new dask config each time? or did you mean only before the worker creation will it work?
what if i use client.restart() e.g. restart workers inside the context manager?
the reason i ask since i do see the new default setting that are applied on the cluster on the time of the creation and doing (no context manager or with give same results)
```auto
dask.config.set(
{
"distributed.worker.memory.spill": 0.1,
"distributed.worker.memory.target": 0.8,
"distributed.worker.memory.pause": 0.9,
"distributed.worker.memory.terminate": 0.95,
"distributed.worker.memory.spill-compression": "auto",
"distributed.scheduler.work-stealing": True,
"distributed.scheduler.worker-saturation": 1.5,
"dataframe.shuffle.method": "p2p",
"dataframe.shuffle.compression": "auto",
"distributed.nanny.environ.MALLOC_TRIM_THRESHOLD_": "1024",
}
)
```
and then logging the `dask.config.config` AFTER the cluster is already setup -i do see the new settings so im wondering here what is going on? (same with `dask.config.get`)
---
### Author:  [@jacobtomlinson](https://dask.discourse.group/u/jacobtomlinson)
#### Post date: [September 2, 2024, 2:29pm UTC](https://dask.discourse.group/t/dask-config-how-does-it-actually-work/3097/6 "2024-09-02T14:29:30Z")
If you’re setting those config options on the client they won’t have any effect on the workers. You need to set them on the workers themselves.
Currently `dask-kubernetes` does not support config forwarding from the client to the workers. If this is a feature that would be valuable to you I recommend you open an issue on the `dask-kubernetes` repo. This would be a great feature to add, but we haven’t had anyone ask for it yet so it hasn’t been implemented.
In the meantime you can set the config as environment variables directly on the workers via the [env kwarg](https://kubernetes.dask.org/en/latest/operator_kubecluster.html#api).
```python
from dask_kubernetes.operator import KubeCluster
cluster = KubeCluster(
name="foo",
image="bar",
env={"DASK_DISTRIBUTED __WORKER__ MEMORY__SPILL": 0.1, ...},
)
```
---
### Author:  [@Hvuj](https://dask.discourse.group/u/Hvuj)
#### Post date: [September 2, 2024, 2:47pm UTC](https://dask.discourse.group/t/dask-config-how-does-it-actually-work/3097/7 "2024-09-02T14:47:01Z")
thank you - yea ill add an issue and yea ill use env vars -
final question:
if i use
```auto
new_config = dask.config.collect(paths=[config_path])
dask.config.update(dask.config.config,new_config)
```
does it go directly to the workers?
---
### Author:  [@jacobtomlinson](https://dask.discourse.group/u/jacobtomlinson)
#### Post date: [September 4, 2024, 4:00pm UTC](https://dask.discourse.group/t/dask-config-how-does-it-actually-work/3097/8 "2024-09-04T16:00:34Z")
No it doesn’t. Unfortunately Dask doesn’t modify config on remote workers at runtime.