Memory calculation: each worker works on each partition?

Hi @henry and welcome to discourse!

Dask can (and will) try to work on as many partitions in parallel on a worker as there are cores available. Therefore, it can actually be better to partition your bag so that many partitions can fit in memory on a worker at a time. There are more details in the Dask Best Practices.

1 Like