How to make the worker fail the computation when memory limit is reached?

5

In terminal:

dask-scheduler

followed by

dask-worker tcp://127.0.0.1:8786 --nprocs 1 --nthreads 1 --memory-limit=200MiB

^Gave the worker a tiny bit of memory to reproduce the issue easier.

The python code

import numpy as np
import pandas as pd
from dask import delayed
from distributed import Client


def f():
    print("running f")
    print("-----")
    print("-----")
    print("-----")
    print("-----")
    df = pd.DataFrame(dict(row_id=np.zeros(10000000)))
    return df


def main():
    with Client(address='tcp://127.0.0.1:8786') as client:
        d_object = delayed(f)()
        print(d_object.compute(scheduler=client))


if __name__ == "__main__":
    main()

The output prints

distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {'f-29283ad1-da89-42db-b065-0bc69f0e7cf7': ('tcp://127.0.0.1:62835',)}
distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {'f-29283ad1-da89-42db-b065-0bc69f0e7cf7': ('tcp://127.0.0.1:63054',)}
distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {'f-29283ad1-da89-42db-b065-0bc69f0e7cf7': ('tcp://127.0.0.1:63058',)}
distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {'f-29283ad1-da89-42db-b065-0bc69f0e7cf7': ('tcp://127.0.0.1:63062',)}
distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {'f-29283ad1-da89-42db-b065-0bc69f0e7cf7': ('tcp://127.0.0.1:63066',)}
distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {'f-29283ad1-da89-42db-b065-0bc69f0e7cf7': ('tcp://127.0.0.1:63073',)}
distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {'f-29283ad1-da89-42db-b065-0bc69f0e7cf7': ('tcp://127.0.0.1:63077',)}
distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {'f-29283ad1-da89-42db-b065-0bc69f0e7cf7': ('tcp://127.0.0.1:63081',)}
distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {'f-29283ad1-da89-42db-b065-0bc69f0e7cf7': ('tcp://127.0.0.1:63085',)}
distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {'f-29283ad1-da89-42db-b065-0bc69f0e7cf7': ('tcp://127.0.0.1:63089',)}
distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {'f-29283ad1-da89-42db-b065-0bc69f0e7cf7': ('tcp://127.0.0.1:63098',)}
1 Like