Collecting Info Logs via Foward Logging

Hello dear Daskers,

I am running Dask via LSFCluster abstraction from the dask_jobqueue library and I am trying to get all worker logs onto the machine triggering the script. After some research, I found the relatively new “forward_logging” method and was intrigued (Related Github PR)

It does seem to work nicely for WARN and ERROR levels but doesn’t forward the INFO levels - would you have any idea which setting I have to tweak for that to be possible?

My minimal test script looks like this where I tried to set all kinds of logging configs to DEBUG:

import logging
import os
import dask
from dask_jobqueue import LSFCluster
from distributed import Client

logging.basicConfig(level=logging.INFO, format='%(message)s')
logger: logging.Logger = logging.getLogger(__name__)

def do_error() -> int:
    logger.error("Hello error")
    return 42

def do_info() -> int:
    logger.warn("Hello warning")"Hello info")  # --> This is not shown
    return 84

if __name__ == "__main__":
    dask.config.set({'logging.loggers.distributed.level': 'DEBUG'})
    dask.config.set({'logging.loggers.distributed.logging.distributed': 'debug'})
    max_memory = 500
    cluster = LSFCluster(
        queue='long',  # short queue for some reason doesn't work properly
        # ncpus=1,
        memory=f'{max_memory}MB',  # Even though it's skipped, it must be set
        # mem=max_memory * 1000000,
        job_extra_directives=[f'-o /tmp/job_out -R "rusage[mem={max_memory}MB/host]"'],  # No Mails, total Memory per node
        job_directives_skip=['-R'],  # Skip other directives about memory
        silence_logs=False  # TODO: Add this for "DEBUG environment"
    compute_client = Client(cluster)
    compute_client.forward_logging()  # forward the root logger at any handled level

    # These work with logging!
    # Nope, info doesn't work

In the example, “Hello error” is shown as well as “Hello warning”, also distributed.core and distributed.scheduler INFO messages are shown, but I can’t get the “Hello info” message to be received. (I also tried compute_client.forward_logging(level=logging.DEBUG) with no avail)

Hi @schulz-m,

I think the problem comes from the fact that the logger object on Worker side isn’t configured to log INFO messages.

As stated in the documentation:

One nuance worth highlighting: even though our client-side root logger is configured with a level of INFO, the worker-side root loggers still have their default level of ERROR because we haven’t done any explicit logging configuration on the workers. Therefore worker-side INFO logs will not be forwarded because they never even get handled in the first place.

It is necessary to set the client-side logger’s level to INFO before the info message will be handled and forwarded to the client. In other words, the “effective” level of the client-side forwarded logging is the maximum of each logger’s client-side and worker-side levels.

Changing your method like below fixes the problem:

def do_info() -> int:
    logger: logging.Logger = logging.getLogger(__name__)
    logger.warn("Hello warning")"Hello info")
    return 84