How to confirm task is being executed in ProcessPool?

Distributed
1

I have a set of transforms which are executed under

    with dask.annotate(executor="processes"):

Is there any way to confirm that the corresponding tasks are being executed in the process pool ? Is even this annotation a soft constraint ?

Annotations are metadata or soft constraints associated with tasks that dask schedulers may choose to respect: They signal intent without enforcing hard constraints. As such, they are primarily designed for use with the distributed scheduler.

https://docs.dask.org/en/latest/api.html#dask.annotate

NOT A CONTRIBUTION

2

Well, according to source code: https://github.com/dask/distributed/blob/main/distributed/worker.py#L2256, I would say that if the task has an executor annotation, it will always use it.

The only problem I could think of is if the annotation was lost during a graph optimization process, but I’m not sure this can happen.

Ultimately, you can check in your WorkerPlugin by implementing the transition function if the tasks have the annotation.

1 Like