My team was trying to upgrade our rapids to 23.12 which requires dask 2023.11, however we were running into the wait_for_workers issue documented here. I notice that this issue has been fixed for distributed >= 2024.1, so we were considering waiting until the release of rapids 24.02 which will hopefully support the fix.
Is there a way to officially check whether rapids 24.02 will support distributed 2024.1? I checked here but it does not pin down a specific version.
I @nickkernan, welcome to Dask community!
Did you try to upgrade Dask/Distributed to 2024.1 version? Since the requirement in Rapids is only to be greater than 2023.11, did you see any reason why it should not work with a more recent version of Distributed?
I assume the nightly Rapids 24.02 would have no issue building with Dask/Distributed 2024.1 now, but I believe when it releases in February it will pin down a specific version (that’s what I’m seeing with stable versions of Rapids, for example 23.12 specifically wants 2023.11.0). I didn’t want it to turn out that Rapids 24.02 would pin down Dask/Distributed 2023.12, for example - then we would need to wait again for the April release.
I suppose I’m just asking if there’s a way to be pretty sure ahead of time which one will be supported.
Okay, I have to admit I don’t know. Maybe @jakirkham or @jacobtomlinson have the answer?
24.02 is pinning to Dask
2024.1.1. We will likely keep the pin the same for
24.04 as it will take a little time for us to adopt the Dask Dataframe changes coming shortly.