What exactly is the bytes stored in the dashboard, and debugging the perf of a simple filter job

Dask Bag
1

I have a simple filter job (based on a value being greater than 0.5) running on about 11 Gibs of data, and no repartioning/shuffling. I’m seeing bytes stored up to 100 GB sometimes during the run, and the job get stuck with the following error message once it has already finished the jobs that writes to disk

Event loop was unresponsive in Nanny for 3.09s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability

The machine has 1 TB of memory.

Also, the dask dashboard often hangs, where the UI is not loading. Wondering if folks might have pointers!

Any ideas? Thanks!

2

Hi @yifanwu,

What kind of data do you have in input? Would it be possible it is 11GiB compressed?

Is the job really stuck? This message can occur depending of your workflow, but it shouldn’t block the job. Anyway, this should not be related to a lack of memory, but more to the computation you perform on the data.

I’ve never seen the Dashboard hang, or could this mean the machine you are using is overloaded?

Would you be able to share any code snippet or even better some reproducer?