How to swap backing store?

Dask Array
1

Is there a simple way to use a different array package in place of NumPy for backing store? Preferably, that would work without having to modify Dask but if not possible then I assume I can always change the NumPy import at the top of each Dask file that has it to import the other array package (that has the same NumPy API. Wanting to start small, I tried importing the different array package in wrap.py and used it in place of np.ones_like in the
ones = array_creation_dispatch.register_inplace(
backend=“numpy”,
name=“ones”,
)(w(broadcast_trick(np.ones_like), dtype=“f8”))

Then I did “python -m pip install -e .” When I try to run a simple example that just does a call to daskarray.ones, I get an error that "cannot import name ‘compute’ from ‘dask’ in dask/array/core.py. I think I’m probably missing something obvious here. Can someone help?

thanks!

2

Update: Normally I’d do “python setup.py develop” but that complains that setup.cfg is missing. I added a modified setup.cfg from another project and then the previous command worked and I no longer get the “cannot import compute” problem.

One thing that I did notice is that in our new backing store that we didn’t support one of the options for ones_like and that this caused a silent failure. For debugging, I started running this with the synchronous scheduler. Is there a debug option that causes all failures to be visible in this mode?

3

Hi @DrTodd13,

Did you see the following page:
https://docs.dask.org/en/stable/how-to/selecting-the-collection-backend.html

?

I think this is what you’re after.

4

Thanks. I think that is what I’m after but could use some more example code.
I did the following but when calling dask.array.ones() rather than getting back a Dask array with my non-NumPy backend, it returns something of my array type directly. Any suggestions? Thanks!

from dask.array.backends import ArrayBackendEntrypoint, array_creation_dispatch
from dask.utils import Dispatch
from dask.array.dispatch import to_numpy_dispatch
from dask.array.core import Array
import my_array_type
import numpy as np

to_mat_dispatch = Dispatch("to_mat_dispatch")

class MatBackendEntrypoint(ArrayBackendEntrypoint):
    @classmethod
    def to_backend_dispatch(cls):
        return to_mat_dispatch

    @classmethod
    def to_backend(cls, data: Array, **kwargs):
        print("to_backend:", type(data), **kwargs)
        if isinstance(data._meta, my_array_type.ndarray):
            return data
        return data.map_blocks(cls.to_backend_dispatch(), **kwargs)

    @staticmethod
    def ones(shape, *, dtype=None, meta=None, **kwargs):
        """Create an array of ones

        Returns a new array having a specified shape and filled
        with ones.
        """
        return my_array_type.ones(shape, dtype=dtype)
5

I agree with you, it’s a bit difficult to really know what to do, but also probably because it’s still new and experimental. I’m afraid this goes far beyond my knowledge.

The only advises I can give you is to continue digging though the source code, or maybe try opening an issue directly in Dask github issue tracker, you might gain more traction here for this complicated problem. In the end, I’m hopping that at least the documentation can be improved.