Problems on running cuml.dask.cluster.KMeans

1

Hi, I have a numpy memmap, how to call cuml.dask.cluster.Kmeans to use the data in the memmap? I convert np memmap to cupy-backed dask array using

darr = dask,array.from_array(memap)
cupy_darr = darr.to_backend(“cupy”)

kmeans = cuml.dask.cluster.KMeans(n_clusters=1000, init_max_iter=1000, oversampling_factor=10)

dist_to_cents = kmeans.fit_transform(cupy_darr)
val = dist_to_cents.compute()

But it failed with the following error message:

<Future: pending, key: test2-3ef3d1ab703f46133a773231e30aba09>
====> (235871, 128) <class ‘dask.array.core.Array’>
2024-05-29 19:42:15,519 - distributed.worker - WARNING - Compute Failed
Key: test2-3ef3d1ab703f46133a773231e30aba09
Function: test2
args: ()
kwargs: {}

Any help is highly appreciated!

2

Hi @fayw, welcome to Dask community!

Would you be able to produce a complete MVCE?

Do you think the problem is linked to memmap, did you try without it?

Do you have a more complete stack trace?