I would like to output the logger of every worker into individual file. In SLURMCluster, there is a parameter log_directory which specifies the path of the log files of workers. But how to do this with LocalCluster?
Any help is appreciated!
Hi @jiwei08, welcome to Dask community!
First, this will not help, but as explained in other places, when using SLURMCluster or any cluster class based on another orchestration level, this is not Dask which manages log files, but the orchestrating system. In the case of SLURMCluster, dask-worker CLI is used under the hood, and outputs all messages to stderr, which are retrieved by Slurm.
AFAIK, there is no easy solution in the LocalCluster case for the time being, because other processed are forked in a clean environment and I’m not sure where the stderr/out are going.
I usually redirect to Debug — Dask documentation, but I just tried to set-up something and failed, I’m afraid this is not up to date. I still think the better chance is to configure logging module properly on every Worker, but I’m not sure of the best way.
There is definitly a room for improvement in the documentation or even in the functionnalities offered by LocalCluster.
Very helpful, thank you!
I think that maybe I can modify the logger handler of each worker to output id of each worker. Then I can identify which worker the log comes from even if the logs are written in the same log file. I will try if this works.
Thanks again!
1 Like
If you achieve anything, please let us know, this would be really nive to share in the documentation!