Problems with object pickle

Distributed
6

Hi @Lori, sorry for the back and forth on this. Upon closer reading of the Serialization - Dask.distributed docs, currently functions are only serialized by pickle and cloud pickle:

Computational tasks like f(x) that are defined and serialized on client processes and deserialized and run on worker processes. These are serialized using a fixed scheme decided on by those libraries. Today this is a combination of pickle and cloudpickle.

Here’s a small python snippet to show this:

from dask.distributed.protocol import serialize
import numpy as np

def my_func(x):
    return len(x)

data = np.arange(20)

# produces an error, "Could not serialize object of type function."
# serialize(my_func, serializers=['dask', 'msgpack'])

s_func = serialize(my_func)
print(f"serializer type: {s_func[0]['serializer']}")
s_data = serialize(data)
print(f"serializer type: {s_data[0]['serializer']}")
2 Likes