NotImplementedError: cannot instantiate 'WindowsPath' on your system

We started using Coiled recently and have been very happy with it. I ran into an issue recently related to pickle and I was hoping someone might be able to shed some light on what might be happening. I’ve been using my corporate Windows laptop to build a simple data pipeline. I can run it locally but when I attempt to run it on my Coiled cluster I get the following exception in distributed/protocol/pickle.py:

 File "venv\lib\site-packages\distributed\client.py", line 289, in _result
    raise exc.with_traceback(tb)
  File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/pickle.py", line 73, in loads
  File "/opt/conda/envs/coiled/lib/python3.9/pathlib.py", line 1084, in __new__
NotImplementedError: cannot instantiate 'WindowsPath' on your system

This exception shows up in the worker logs so it appears that the client, which is my Windows laptop, is pickling something using WindowsPath. This causes a problem on the workers which are running on a Linux container. I am using Prefect to execute this code. The task function that fails has a single line of code that returns an empty Pandas dataframe. Happy to share the entire stack trace if needed but the pipeline is dead simple. I’ve just been trying to confirm the pipeline runs successfully on dask-distributed (Coiled).

Thanks in advance for your help.

Hi,
Yes, the problem is that your laptop runs on Windows while Coiled workers run on Linux. You’re experiencing a bug; it’s supposed to work transparently.

Without any idea on what algorithm you are trying to run, however, there’s little help we can offer.
The stack trace won’t really help, as it will just say that unpickle fails when the worker tries to run your task.

Could you open a ticket with a minimal reproducer on GitHub - dask/distributed: A distributed task scheduler for Dask?

The issue is being tracked at NotImplementedError: cannot instantiate 'WindowsPath' on your system · Issue #7460 · dask/distributed · GitHub