Is Dask aware of other jobs running on a server besides its own? Does dask-scheduler take system load caused by other running processes (besides it’s own) into account when it giving tasks to a worker on a machine?
@andreas-nt Welcome to Discourse, and thanks for your question!
Broadly speaking, Dask isn’t aware of exact computations but can infer that it has less memory on a particular worker than the maximum capacity, see unmanaged memory here: Worker Memory Management — Dask.distributed 2022.8.1 documentation
Could you please share more details about your workflow here? It’ll allow us to help you better. 