Is Dask aware of other jobs running on the system?

Is Dask aware of other jobs running on a server besides its own? Does dask-scheduler take system load caused by other running processes (besides it’s own) into account when it giving tasks to a worker on a machine?

@andreas-nt Welcome to Discourse, and thanks for your question!

Broadly speaking, Dask isn’t aware of exact computations but can infer that it has less memory on a particular worker than the maximum capacity, see unmanaged memory here: Worker Memory Management — Dask.distributed 2022.4.0 documentation

Could you please share more details about your workflow here? It’ll allow us to help you better. :smile: