We started using Coiled recently and have been very happy with it. I ran into an issue recently related to pickle and I was hoping someone might be able to shed some light on what might be happening. I’ve been using my corporate Windows laptop to build a simple data pipeline. I can run it locally but when I attempt to run it on my Coiled cluster I get the following exception in distributed/protocol/pickle.py:
File "venv\lib\site-packages\distributed\client.py", line 289, in _result
raise exc.with_traceback(tb)
File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/pickle.py", line 73, in loads
File "/opt/conda/envs/coiled/lib/python3.9/pathlib.py", line 1084, in __new__
NotImplementedError: cannot instantiate 'WindowsPath' on your system
This exception shows up in the worker logs so it appears that the client, which is my Windows laptop, is pickling something using WindowsPath. This causes a problem on the workers which are running on a Linux container. I am using Prefect to execute this code. The task function that fails has a single line of code that returns an empty Pandas dataframe. Happy to share the entire stack trace if needed but the pipeline is dead simple. I’ve just been trying to confirm the pipeline runs successfully on dask-distributed (Coiled).
Thanks in advance for your help.