Hallo,
I have scoured most of the docs related to deployments of distributed computing solutions, and it seems that Dask is the way to go. However, my setup would require to abstract Kubernetes away from the analysts. Which makes the dask-gateway very interesting for us.
I have followed the documentation ( Install on a Kubernetes Cluster — Dask Gateway 2025.4.1-0.dev documentation ) on installing the gateway server, I will not need the Jupyter hub integration since we use uv and Marimo for our development environments.
I have come so far as to have the cluster up and running and have workers configured with the scale method or the adapt method successfully.
However, I seem to run into a very old (last post 2022) issue:
I tried to downgrade the distributed and dask packages, but there is a new dependency on the dask gateway version:
Install dependency error
Because dask-gateway==2025.4.0 depends on distributed>=2022.4.0 and you require dask-gateway==2025.4.0, we can conclude
that you require distributed>=2022.4.0.
And because you require distributed==2.30.0, we can conclude that your requirements are unsatisfiable.
Is there a trivial solution to this, or is this more a fact that support for dask-gateway is very limited and seems like support for this product will likely be terminated entirely in the future?
Looking at the open issues on github, it seems support for hadoop will be dropped, leaving only Kubernetes or local deployment as an option. (slurm pipelines are also failing atm)
This can be an interesting project to pick up again due to the recent developments on python environment management and Marimo coming to the party with a great alternative to notebooks. (dask-gatewaycan potentially be updated to use the anywidget instead of ipywidgets since Jupyterhub also supports anywidget)
Edit:
I have managed to get a client connected by using version:
dask-gateway==2024.1.0