What can be causes for Semaphore initialization errors

Distributed
1

I am using a semaphore to control access and I see that it very frequently ends up in the state

RuntimeError: <class 'distributed.semaphore.Semaphore'> object not properly initialized. This can happen if the object is being deserialized outside of the context of a Client or Worker.

When using smaller worker pools for testing e.g. 20 nodes, I haven’t seen this issue. I am curious what situations can lead to this improper initialization.

Interestingly I notice these errors even when the worker in question has actually successfully used the semaphore for a couple of tasks.

I see that the related code changes were made a couple of months back: Future deserialization without available client by fjetter · Pull Request #7580 · dask/distributed · GitHub

NOT A CONTRIBUTION

2

Hi @vij,

I honestly don’t know what could cause this error. Pinging @fjetter in case he can identify a reason.

If this is problematic for you and we don’t get an answer in the following days, I recomment opening a github issue, with if possible an example code/workflow.

3

Thanks.

Will try to create a minimal reproduction in the mean time.

4

An interesting observation. I was able to consistently get these failures when the lease acquisition was attempted in a ProcessPool. i.e.,

        with dask.annotate(executor="processes"):
            <code which uses the semaphore>

the RuntimeError occurs consistently. However I am also seeing these errors in other scenarios. Still debugging these.

NOT A CONTRIBUTION

5

I would classify such a RuntimeError as a bug and would ask you to open a ticket on the distributed tracker with a minimal reproducer and the distributed version used.
We had a similar looking issue shortly after the above linked PR was merged. I’m curious if this still exists in the most recent version.

get these failures when the lease acquisition was attempted in a ProcessPool . i.e.,

Yeah. ProcessPools are a bit odd and not properly supported for many things (one of the reasons why they are a couple of manual steps required to even install them).
Semaphores currently require a Worker or Client to be present in the same process. You’d need to ensure this using an initializer

6

Thanks for confirming. I will follow up with a ticket.