Collecting Info Logs via Foward Logging

Distributed
1

Hello dear Daskers,

I am running Dask via LSFCluster abstraction from the dask_jobqueue library and I am trying to get all worker logs onto the machine triggering the script. After some research, I found the relatively new “forward_logging” method and was intrigued (Related Github PR)

It does seem to work nicely for WARN and ERROR levels but doesn’t forward the INFO levels - would you have any idea which setting I have to tweak for that to be possible?

My minimal test script looks like this where I tried to set all kinds of logging configs to DEBUG:

import logging
import os
import dask
from dask_jobqueue import LSFCluster
from distributed import Client


logging.basicConfig(level=logging.INFO, format='%(message)s')
logger: logging.Logger = logging.getLogger(__name__)


def do_error() -> int:
    logger.error("Hello error")
    return 42

def do_info() -> int:
    logger.warn("Hello warning")
    logger.info("Hello info")  # --> This is not shown
    return 84

if __name__ == "__main__":
    dask.config.set({'logging.loggers.distributed.level': 'DEBUG'})
    dask.config.set({'logging.loggers.distributed.logging.distributed': 'debug'})
    os.environ['DASK_LOGGING__DISTRIBUTED'] = 'DEBUG'
    
    max_memory = 500
    cluster = LSFCluster(
        queue='long',  # short queue for some reason doesn't work properly
        cores=1,
        # ncpus=1,
        processes=1,
        memory=f'{max_memory}MB',  # Even though it's skipped, it must be set
        # mem=max_memory * 1000000,
        walltime='72:00',
        job_extra_directives=[f'-o /tmp/job_out -R "rusage[mem={max_memory}MB/host]"'],  # No Mails, total Memory per node
        job_directives_skip=['-R'],  # Skip other directives about memory
        silence_logs=False  # TODO: Add this for "DEBUG environment"
    )
    
    cluster.scale(1)
    
    compute_client = Client(cluster)
    compute_client.forward_logging()  # forward the root logger at any handled level

    # These work with logging!
    print(compute_client.submit(do_error).result())
    
    # Nope, info doesn't work
    print(compute_client.submit(do_info).result())

In the example, “Hello error” is shown as well as “Hello warning”, also distributed.core and distributed.scheduler INFO messages are shown, but I can’t get the “Hello info” message to be received. (I also tried compute_client.forward_logging(level=logging.DEBUG) with no avail)

2

Hi @schulz-m,

I think the problem comes from the fact that the logger object on Worker side isn’t configured to log INFO messages.

As stated in the documentation:

One nuance worth highlighting: even though our client-side root logger is configured with a level of INFO, the worker-side root loggers still have their default level of ERROR because we haven’t done any explicit logging configuration on the workers. Therefore worker-side INFO logs will not be forwarded because they never even get handled in the first place.

It is necessary to set the client-side logger’s level to INFO before the info message will be handled and forwarded to the client. In other words, the “effective” level of the client-side forwarded logging is the maximum of each logger’s client-side and worker-side levels.

Changing your method like below fixes the problem:

def do_info() -> int:
    logger: logging.Logger = logging.getLogger(__name__)
    logger.setLevel(logging.INFO)
    logger.warn("Hello warning")
    logger.info("Hello info")
    return 84