Disable dask worker logs from printing on console and write to file

Distributed
4

Hi @SOUMYASHUKLA, thanks for the question! To start, Distributed uses Python’s standard logging module and you can always check the current configuration settings using dask.config.get. To answer your questions:

To only print error messages (see this similar discourse question), you can change the logging level in ~/.config/dask/*.yaml (see controlling logging via a config file). For example, your ~/.config/dask/distributed.yaml file could look like:

logging:
  distributed: error

If you prefer to temporarily set this directly within Python you can use:

import dask
dask.config.set({'logging.distributed': 'error'})

In the snippet you shared from the issue for adding documentation on saving logs to a file, you can change the logging level such that only errors are saved and/or printed to the console, as shown above.

I was also wondering if the warning or INFO message can be logged once (maybe with a counter for occurrences).

I would recommend controlling this using a filter. There are examples on how to do this in the logging-cookbook, and then you’d change the config file to something like this, where custom_filter_class is defined in a separate Python file:

logging:
  version: 1
  handlers:
    file:
      class: logging.handlers.RotatingFileHandler
      filename: output.log
      level: INFO
      filters: [custom_filter_class]
  loggers:
    distributed:
      level: INFO
      handlers:
        - file
4 Likes