I’m trying to profile my single and multi-threaded dask calls (dask.config.set(scheduler='threads')
etc.) for some simple DataFrame operations on a 10gb collection of files. I was hoping just to get some simple log info on when each operation begins and ends but my calls to logging.basicConfig(level=logging.INFO) or logging.basicConfig(level=logging.DEBUG) does not produce any dask-related output. Am I missing a step or is this just not supported. I’m using dask version 2023.10.0 on python 3.10.2.
Hi @spillz, welcome to Dask community!
I don’t think that Dask logs things like that by itself.
However, depending on what you want, you have some other tools available, see
https://docs.dask.org/en/latest/how-to/debug.html
and
https://docs.dask.org/en/latest/diagnostics-local.html
in particular.
Many thanks! It looks like ProgressBar and Profiler can do what I need.