Client and Worker with different python versions

Distributed
1

Hello! I’m trying to run the example here Client — Dask.distributed 2022.12.1 documentation with python 3.7 client and python 3.9 worker.
I installed pickle5 to mitigate the pickling protocol change in python 3.8 and I’m also using dask and distributed 2021.11.2 which supports python 3.7 (even though it’s running on top of python 3.9).
I’m experiencing a bug that seems to be related to cloudpickle TypeError: an integer is required (got type bytes) · Issue #404 · cloudpipe/cloudpickle · GitHub
I also encountered issue #1678 on distributed (I’m blocked from linking more than 2 pages) warning about mismatching versions. Does it mean dask does not support such discrepancy or should there be a way to resolve the issue?
Thanks!
Here is a stack trace from the worker:

distributed.worker - WARNING - Mismatched versions found
+---------+----------------+----------------+-------------------------------------+
| Package | This Worker    | scheduler      | workers                             |
+---------+----------------+----------------+-------------------------------------+
| blosc   | 1.11.1         | 1.11.1         | {'1.7.0', '1.11.1'}                 |
| lz4     | 3.1.10         | 3.1.10         | {'3.1.10', '3.1.0'}                 |
| msgpack | 1.0.4          | 1.0.4          | {'1.0.3', '1.0.4'}                  |
| numpy   | 1.22.4         | 1.22.4         | {'1.19.2', '1.22.4'}                |
| pandas  | 1.4.4          | 1.4.4          | {'1.3.5', '1.4.4'}                  |
| python  | 3.9.13.final.0 | 3.9.13.final.0 | {'3.9.13.final.0', '3.7.6.final.0'} |
| toolz   | 0.12.0         | 0.12.0         | {'0.12.0', '0.11.2'}                |
| tornado | 6.2            | 6.2            | {'6.1', '6.2'}                      |
+---------+----------------+----------------+-------------------------------------+
Notes:
-  msgpack: Variation is ok, as long as everything is above 0.6
distributed.worker - INFO -         Registered to: tcp://mlrun-iqjxz-func-gda-9fb940e2-4.default-tenant:8786
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.protocol.pickle - INFO - Failed to deserialize b'\x80\x04\x95\xd7\x01\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x0e_make_function\x94\x93\x94(h\x00\x8c\r_builtin_type\x94\x93\x94\x8c\x08CodeType\x94\x85\x94R\x94(K\x01K\x00K\x01K\x02KCC\x10t\x00d\x01\x83\x01\x01\x00|\x00d\x02\x17\x00S\x00\x94N\x8c\x06incing\x94K\x02\x87\x94\x8c\x05print\x94\x85\x94\x8c\x01x\x94\x85\x94\x8c\x1e<ipython-input-1-5d8ce15efc6a>\x94\x8c\x03inc\x94K\x01C\x04\x00\x01\x08\x01\x94))t\x94R\x94}\x94(\x8c\x0b__package__\x94N\x8c\x08__name__\x94\x8c\x08__main__\x94uNNNt\x94R\x94\x8c\x1ccloudpickle.cloudpickle_fast\x94\x8c\x12_function_setstate\x94\x93\x94h\x19}\x94}\x94(h\x16h\x10\x8c\x0c__qualname__\x94h\x10\x8c\x0f__annotations__\x94}\x94\x8c\x0e__kwdefaults__\x94N\x8c\x0c__defaults__\x94N\x8c\n__module__\x94h\x17\x8c\x07__doc__\x94N\x8c\x0b__closure__\x94N\x8c\x17_cloudpickle_submodules\x94]\x94\x8c\x0b__globals__\x94}\x94u\x86\x94\x86R0.'
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/distributed/worker.py", line 4220, in loads_function
    result = cache_loads[bytes_object]
  File "/opt/conda/lib/python3.9/site-packages/distributed/utils.py", line 1363, in __getitem__
    value = super().__getitem__(key)
  File "/opt/conda/lib/python3.9/collections/__init__.py", line 1058, in __getitem__
    raise KeyError(key)
KeyError: b'\x80\x04\x95\xd7\x01\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x0e_make_function\x94\x93\x94(h\x00\x8c\r_builtin_type\x94\x93\x94\x8c\x08CodeType\x94\x85\x94R\x94(K\x01K\x00K\x01K\x02KCC\x10t\x00d\x01\x83\x01\x01\x00|\x00d\x02\x17\x00S\x00\x94N\x8c\x06incing\x94K\x02\x87\x94\x8c\x05print\x94\x85\x94\x8c\x01x\x94\x85\x94\x8c\x1e<ipython-input-1-5d8ce15efc6a>\x94\x8c\x03inc\x94K\x01C\x04\x00\x01\x08\x01\x94))t\x94R\x94}\x94(\x8c\x0b__package__\x94N\x8c\x08__name__\x94\x8c\x08__main__\x94uNNNt\x94R\x94\x8c\x1ccloudpickle.cloudpickle_fast\x94\x8c\x12_function_setstate\x94\x93\x94h\x19}\x94}\x94(h\x16h\x10\x8c\x0c__qualname__\x94h\x10\x8c\x0f__annotations__\x94}\x94\x8c\x0e__kwdefaults__\x94N\x8c\x0c__defaults__\x94N\x8c\n__module__\x94h\x17\x8c\x07__doc__\x94N\x8c\x0b__closure__\x94N\x8c\x17_cloudpickle_submodules\x94]\x94\x8c\x0b__globals__\x94}\x94u\x86\x94\x86R0.'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/distributed/protocol/pickle.py", line 75, in loads
    return pickle.loads(x)
TypeError: an integer is required (got type bytes)
distributed.worker - ERROR - Could not deserialize task
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/distributed/worker.py", line 4220, in loads_function
    result = cache_loads[bytes_object]
  File "/opt/conda/lib/python3.9/site-packages/distributed/utils.py", line 1363, in __getitem__
    value = super().__getitem__(key)
  File "/opt/conda/lib/python3.9/collections/__init__.py", line 1058, in __getitem__
    raise KeyError(key)
KeyError: b'\x80\x04\x95\xd7\x01\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x0e_make_function\x94\x93\x94(h\x00\x8c\r_builtin_type\x94\x93\x94\x8c\x08CodeType\x94\x85\x94R\x94(K\x01K\x00K\x01K\x02KCC\x10t\x00d\x01\x83\x01\x01\x00|\x00d\x02\x17\x00S\x00\x94N\x8c\x06incing\x94K\x02\x87\x94\x8c\x05print\x94\x85\x94\x8c\x01x\x94\x85\x94\x8c\x1e<ipython-input-1-5d8ce15efc6a>\x94\x8c\x03inc\x94K\x01C\x04\x00\x01\x08\x01\x94))t\x94R\x94}\x94(\x8c\x0b__package__\x94N\x8c\x08__name__\x94\x8c\x08__main__\x94uNNNt\x94R\x94\x8c\x1ccloudpickle.cloudpickle_fast\x94\x8c\x12_function_setstate\x94\x93\x94h\x19}\x94}\x94(h\x16h\x10\x8c\x0c__qualname__\x94h\x10\x8c\x0f__annotations__\x94}\x94\x8c\x0e__kwdefaults__\x94N\x8c\x0c__defaults__\x94N\x8c\n__module__\x94h\x17\x8c\x07__doc__\x94N\x8c\x0b__closure__\x94N\x8c\x17_cloudpickle_submodules\x94]\x94\x8c\x0b__globals__\x94}\x94u\x86\x94\x86R0.'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/distributed/worker.py", line 3275, in _maybe_deserialize_task
    function, args, kwargs = _deserialize(*ts.runspec)
  File "/opt/conda/lib/python3.9/site-packages/distributed/worker.py", line 4231, in _deserialize
    function = loads_function(function)
  File "/opt/conda/lib/python3.9/site-packages/distributed/worker.py", line 4222, in loads_function
    result = pickle.loads(bytes_object)
  File "/opt/conda/lib/python3.9/site-packages/distributed/protocol/pickle.py", line 75, in loads
    return pickle.loads(x)
TypeError: an integer is required (got type bytes)
distributed.worker - ERROR - Exception during execution of task inc-f731ad58781b1c0f2467984a44a0e6dd.
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/distributed/worker.py", line 4220, in loads_function
    result = cache_loads[bytes_object]
  File "/opt/conda/lib/python3.9/site-packages/distributed/utils.py", line 1363, in __getitem__
    value = super().__getitem__(key)
  File "/opt/conda/lib/python3.9/collections/__init__.py", line 1058, in __getitem__
    raise KeyError(key)
KeyError: b'\x80\x04\x95\xd7\x01\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x0e_make_function\x94\x93\x94(h\x00\x8c\r_builtin_type\x94\x93\x94\x8c\x08CodeType\x94\x85\x94R\x94(K\x01K\x00K\x01K\x02KCC\x10t\x00d\x01\x83\x01\x01\x00|\x00d\x02\x17\x00S\x00\x94N\x8c\x06incing\x94K\x02\x87\x94\x8c\x05print\x94\x85\x94\x8c\x01x\x94\x85\x94\x8c\x1e<ipython-input-1-5d8ce15efc6a>\x94\x8c\x03inc\x94K\x01C\x04\x00\x01\x08\x01\x94))t\x94R\x94}\x94(\x8c\x0b__package__\x94N\x8c\x08__name__\x94\x8c\x08__main__\x94uNNNt\x94R\x94\x8c\x1ccloudpickle.cloudpickle_fast\x94\x8c\x12_function_setstate\x94\x93\x94h\x19}\x94}\x94(h\x16h\x10\x8c\x0c__qualname__\x94h\x10\x8c\x0f__annotations__\x94}\x94\x8c\x0e__kwdefaults__\x94N\x8c\x0c__defaults__\x94N\x8c\n__module__\x94h\x17\x8c\x07__doc__\x94N\x8c\x0b__closure__\x94N\x8c\x17_cloudpickle_submodules\x94]\x94\x8c\x0b__globals__\x94}\x94u\x86\x94\x86R0.'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/distributed/worker.py", line 3356, in execute
    function, args, kwargs = await self._maybe_deserialize_task(
  File "/opt/conda/lib/python3.9/site-packages/distributed/worker.py", line 3275, in _maybe_deserialize_task
    function, args, kwargs = _deserialize(*ts.runspec)
  File "/opt/conda/lib/python3.9/site-packages/distributed/worker.py", line 4231, in _deserialize
    function = loads_function(function)
  File "/opt/conda/lib/python3.9/site-packages/distributed/worker.py", line 4222, in loads_function
    result = pickle.loads(bytes_object)
  File "/opt/conda/lib/python3.9/site-packages/distributed/protocol/pickle.py", line 75, in loads
    return pickle.loads(x)
TypeError: an integer is required (got type bytes)
2

Hi @AlonMaor14,

In most cases, having different versions of Python, Dask or any upstream packages between Scheduler/Workers/Client is highly discouraged. This may lead to the issue your facing, hence the warning. Here you have a big difference in Python versions, which is bad.

You might or might not find a solution to your problem by trying to get the correct library version on you client, but it sounds like you already tried.