My team was trying to upgrade our rapids to 23.12 which requires dask 2023.11, however we were running into the wait_for_workers issue documented here. I notice that this issue has been fixed for distributed >= 2024.1, so we were considering waiting until the release of rapids 24.02 which will hopefully support the fix.
Is there a way to officially check whether rapids 24.02 will support distributed 2024.1? I checked here but it does not pin down a specific version.
Thanks!
I @nickkernan, welcome to Dask community!
Did you try to upgrade Dask/Distributed to 2024.1 version? Since the requirement in Rapids is only to be greater than 2023.11, did you see any reason why it should not work with a more recent version of Distributed?
Thank you!
I assume the nightly Rapids 24.02 would have no issue building with Dask/Distributed 2024.1 now, but I believe when it releases in February it will pin down a specific version (that’s what I’m seeing with stable versions of Rapids, for example 23.12 specifically wants 2023.11.0). I didn’t want it to turn out that Rapids 24.02 would pin down Dask/Distributed 2023.12, for example - then we would need to wait again for the April release.
I suppose I’m just asking if there’s a way to be pretty sure ahead of time which one will be supported.
RAPIDS 24.02 is pinning to Dask 2024.1.1. We will likely keep the pin the same for 24.04 as it will take a little time for us to adopt the Dask Dataframe changes coming shortly.