Connection between 32 and 64 bits platforms

Distributed
1

I have a Dask cluster run on python 3.9 32bit platform which I try to connect from python 3.9 64 bit on another machine and got the following error:

...
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 3.62 EiB for an array with shape (4179340454199820288,) and data type uint8

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/foa_python_venv/lib/python3.9/site-packages/distributed/client.py", line 942, in __init__
    self.start(timeout=timeout)
  File "/opt/foa_python_venv/lib/python3.9/site-packages/distributed/client.py", line 1100, in start
    sync(self.loop, self._start, **kwargs)
  File "/opt/foa_python_venv/lib/python3.9/site-packages/distributed/utils.py", line 385, in sync
    raise exc.with_traceback(tb)
  File "/opt/foa_python_venv/lib/python3.9/site-packages/distributed/utils.py", line 358, in f
    result = yield future
  File "/opt/foa_python_venv/lib/python3.9/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/opt/foa_python_venv/lib/python3.9/site-packages/distributed/client.py", line 1192, in _start
    await self._ensure_connected(timeout=timeout)
  File "/opt/foa_python_venv/lib/python3.9/site-packages/distributed/client.py", line 1255, in _ensure_connected
    comm = await connect(
  File "/opt/foa_python_venv/lib/python3.9/site-packages/distributed/comm/core.py", line 331, in connect
    raise OSError(
OSError: Timed out during handshake while connecting to tcp://192.168.1.14:8786 after 30 s

Is there any way to connect both platforms?
TIA

2

My exploration showed that the issue is not bitness of the connecting platforms but byteorder - the client has little byteorder, whereas the scheduler has big one. I haven’t found any options for client or scheduler influencing endianess so I think it’s not possible. Anyway I don’t think I’m not the only one who ran into such a problem, so are there any plans to support the work of the distributed with different byteorders?

3

@bulklodd Welcome to Discourse!

The error message checks out, 32-bit machines have a maximum of ~4GB RAM, and it looks like the numpy array is hitting this memory limit. Edit: See Guido’s message below!

I looked at this with @ian, and we’d be curious to see if even accessing the cluster from a 32-bit client would actually work?

Is there any way to connect both platforms?
…
are there any plans to support the work of the distributed with different byteorders?

We don’t think so, because it’s a very niche scenario.

@crusaderky do you have thoughts on this?

4

32-bit machines can allocate either 2GiB or 3GiB of memory per process, depending on kernel configuration.

The error message, however, complains about being unable to allocate 3.6 EiB (exabytes), or a bit less than FOUR MILLION TERABYTES - so something else is going on.
I would check for a “fat finger” error to begin with.

In theory, a 32 bit client connecting to a 64 bit server could work.
In practice, many numpy data types (like int) default to 32 or 64 bits respectively on the two architectures, so I expect many bugs. There are no tests whatsoever on such a mixed configuration, and writing them would not be straightforward either.

1 Like