I did attempt scatter for a similar purpose and I noticed that
- Same object seems to be transferred several times to a worker for a single
compute
call. - The object definitely is re-transferred across
compute
calls.
Available options for collective communication in Dask - #9 by vij
I am curious if there is a mechanism to specify during the scatter that the object needs to be preserved in the a local cache for future use and when the map
calls are made the worker checks for the required key in local-cache
before fetching it from other workers.