i am working on use case to store progress of each task and during research found the plugins are better suited such use case, but it turns out Redis objects are not serialised properly and getting the
distributed.protocol.pickle - ERROR - Failed to serialize (which has redis connection ).
is there a way to serialise the Redis object in generation any object which are by default not supported
Hi @prashant, welcome to Dask community!
Scheduler plugin seems indeed to be the correct approach here, are you doing something similar to the RabbitMQ example? Could you post a code snippet of what you are doing and when you are getting the error?
If you instanciate the Reddis connection at the right time, it shouldn’t have to be serialized.