Logging info available for dask DataFrame compute or to_parquet calls?

Dask DataFrame
1

I’m trying to profile my single and multi-threaded dask calls (dask.config.set(scheduler='threads') etc.) for some simple DataFrame operations on a 10gb collection of files. I was hoping just to get some simple log info on when each operation begins and ends but my calls to logging.basicConfig(level=logging.INFO) or logging.basicConfig(level=logging.DEBUG) does not produce any dask-related output. Am I missing a step or is this just not supported. I’m using dask version 2023.10.0 on python 3.10.2.

2

Hi @spillz, welcome to Dask community!

I don’t think that Dask logs things like that by itself.

However, depending on what you want, you have some other tools available, see
https://docs.dask.org/en/latest/how-to/debug.html

and
https://docs.dask.org/en/latest/diagnostics-local.html

in particular.

3

Many thanks! It looks like ProgressBar and Profiler can do what I need.