# Deploying Dask **URL:** https://dask.discourse.group/c/deploying-dask/9.md [Latest](https://dask.discourse.group/latest.md) · [Categories](https://dask.discourse.group/categories.md) · [Tags](https://dask.discourse.group/tags.md) --- ## [About the Deploying Dask category](https://dask.discourse.group/t/about-the-deploying-dask-category/17)
**Author:** [@ian](https://dask.discourse.group/u/ian)\ **Replies:** 0
Discussions about how to deploy Dask in various contexts --- ## [Use lsf with dask with two workers classes](https://dask.discourse.group/t/use-lsf-with-dask-with-two-workers-classes/4154)
**Author:** [@vianneyl](https://dask.discourse.group/u/vianneyl)\ **Replies:** 2\ **Last updated:** [February 8, 2026, 4:01pm UTC](https://dask.discourse.group/t/use-lsf-with-dask-with-two-workers-classes/4154 "2026-02-08T16:01:49Z")
Hi, I am currently running a cluster on my local machine. It is made up of standard dask workers and dask-cuda workers (both instantiated from terminal, as well as scheduler). Basically it is equivalent to a Spec luster… --- ## [Dask-Gateway 2025/26](https://dask.discourse.group/t/dask-gateway-2025-26/4137)
**Author:** [@lyfo-dev](https://dask.discourse.group/u/lyfo-dev)\ **Replies:** 1\ **Last updated:** [December 15, 2025, 4:15pm UTC](https://dask.discourse.group/t/dask-gateway-2025-26/4137 "2025-12-15T16:15:48Z")
Hallo, I have scoured most of the docs related to deployments of distributed computing solutions, and it seems that Dask is the way to go. However, my setup would require to abstract Kubernetes away from the analysts. W… --- ## [Dask Kubernetes / Memory issue with scheduler POD](https://dask.discourse.group/t/dask-kubernetes-memory-issue-with-scheduler-pod/4088)
**Author:** [@Hugo54](https://dask.discourse.group/u/Hugo54)\ **Replies:** 4\ **Last updated:** [October 2, 2025, 12:53pm UTC](https://dask.discourse.group/t/dask-kubernetes-memory-issue-with-scheduler-pod/4088 "2025-10-02T12:53:11Z")
Hi, After deploying dask kubernetes I had an issue with a user due to a big memory usage at the “gathering” step. We understood in the case of Dask Kubernetes deployement that the scheduler gather all the data in the PO… --- ## [Starting slurm based dask cluster using pre-allocated resources](https://dask.discourse.group/t/starting-slurm-based-dask-cluster-using-pre-allocated-resources/3911)
**Author:** [@maneesh29s](https://dask.discourse.group/u/maneesh29s)\ **Replies:** 7\ **Last updated:** [September 13, 2025, 7:24am UTC](https://dask.discourse.group/t/starting-slurm-based-dask-cluster-using-pre-allocated-resources/3911 "2025-09-13T07:24:50Z")
In our usecase, our application gets a set of pre-allocated resources (nodes, cpus, ram) which we ask for when starting application with sbatch or salloc. Our requirement is that the application should run within those r… --- ## [Conflict with dask and tlz in napari](https://dask.discourse.group/t/conflict-with-dask-and-tlz-in-napari/2663)
**Author:** [@BornKrefeld](https://dask.discourse.group/u/BornKrefeld)\ **Replies:** 5\ **Last updated:** [June 24, 2025, 3:17am UTC](https://dask.discourse.group/t/conflict-with-dask-and-tlz-in-napari/2663 "2025-06-24T03:17:20Z")
I would say I’m a senior coder. My challenge is I used an example of imread dask in image reading napari with dask version 2024.2.1. in pycharm. There is always an error with tlz. My example was derived from tutorial nap… --- ## [Dask Gateway on EKS running Karpenter](https://dask.discourse.group/t/dask-gateway-on-eks-running-karpenter/3980)
**Author:** [@vish727](https://dask.discourse.group/u/vish727)\ **Replies:** 4\ **Last updated:** [June 20, 2025, 3:28pm UTC](https://dask.discourse.group/t/dask-gateway-on-eks-running-karpenter/3980 "2025-06-20T15:28:19Z")
Just curious, has anyone been able to get a proper dask gateway service to run on EKS with karpenter powered autoscaling? I have been trying for almost a year, and while it does work at smaller scale, once we try to get … --- ## [Missing runner in Dask job](https://dask.discourse.group/t/missing-runner-in-dask-job/3900)
**Author:** [@Sam](https://dask.discourse.group/u/Sam)\ **Replies:** 3\ **Last updated:** [April 16, 2025, 4:43pm UTC](https://dask.discourse.group/t/missing-runner-in-dask-job/3900 "2025-04-16T16:43:32Z")
Hi We used Flyte to create Dask task to deploy Dask job on k8s. Generally, it works fine, but sometimes we encountered the Dask scheduler and workers are created and idle. But the runner was not found. Any idea of such i… --- ## [Dask Gateway 2025 release](https://dask.discourse.group/t/dask-gateway-2025-release/3888)
**Author:** [@mapsacosta](https://dask.discourse.group/u/mapsacosta)\ **Replies:** 6\ **Last updated:** [April 10, 2025, 7:24pm UTC](https://dask.discourse.group/t/dask-gateway-2025-release/3888 "2025-04-10T19:24:26Z")
Dear Dask-Gateway devs, Do you have an estimated release date for the 2025.2.0 prod version of dask-gateway? We have been running Dask Gateway (a modified version of it) which integrates with HTCondor and COFFEA (A CERN… --- ## [Jobqueue over ssh](https://dask.discourse.group/t/jobqueue-over-ssh/3757)
**Author:** [@akhmerov](https://dask.discourse.group/u/akhmerov)\ **Replies:** 3\ **Last updated:** [January 3, 2025, 2:28pm UTC](https://dask.discourse.group/t/jobqueue-over-ssh/3757 "2025-01-03T14:28:18Z")
I would like to streamline deploying dask on the following infrastructure. My users have access to a jupyterhub and an HPC cluster. The HPC cluster is only visible from the jupyterhub via an SSH proxy. I would like the C… --- ## [GPCCluster scaling number of VMs](https://dask.discourse.group/t/gpccluster-scaling-number-of-vms/3745)
**Author:** [@bhofner](https://dask.discourse.group/u/bhofner)\ **Replies:** 3\ **Last updated:** [December 16, 2024, 1:18pm UTC](https://dask.discourse.group/t/gpccluster-scaling-number-of-vms/3745 "2024-12-16T13:18:51Z")
Hello, I have read through the Dask Cloud Provider docs and tried to understand how to influence or control the number of VMs the cluster spins up or down. GPCCluster.scale(n\_workers, mem, cores) seems to be the right… --- ## [It is possible to disable autoscaling behavior on a dask job](https://dask.discourse.group/t/it-is-possible-to-disable-autoscaling-behavior-on-a-dask-job/3739)
**Author:** [@u\_ser](https://dask.discourse.group/u/u_ser)\ **Replies:** 1\ **Last updated:** [December 11, 2024, 5:24pm UTC](https://dask.discourse.group/t/it-is-possible-to-disable-autoscaling-behavior-on-a-dask-job/3739 "2024-12-11T17:24:18Z")
For some workloads the behaviour of starting to calculate before the cluster is “complete” (before it has the replicas size number of nodes) is causing to us some problems, would be possible to disable this behaviour on… --- ## [Dask-kubernetes cluster (custom resource) in Pending](https://dask.discourse.group/t/dask-kubernetes-cluster-custom-resource-in-pending/3677)
**Author:** [@Reid\_Pinchback](https://dask.discourse.group/u/Reid_Pinchback)\ **Replies:** 4\ **Last updated:** [November 19, 2024, 5:45pm UTC](https://dask.discourse.group/t/dask-kubernetes-cluster-custom-resource-in-pending/3677 "2024-11-19T17:45:48Z")
I have successfully deployed a Dask k8s cluster via the operator helm chart and used the example custom resource (cluster-spec.yml) from Migrating from classic — Dask Kubernetes 2024.9.1.dev4+gf30da72 documentation. Cli… --- ## [Setup of Dask on HPC](https://dask.discourse.group/t/setup-of-dask-on-hpc/3636)
**Author:** [@Sree\_Harsha](https://dask.discourse.group/u/Sree_Harsha)\ **Replies:** 3\ **Last updated:** [November 8, 2024, 3:49pm UTC](https://dask.discourse.group/t/setup-of-dask-on-hpc/3636 "2024-11-08T15:49:09Z")
Hello all, I installed Dask on an HPC. There was a dependency of mpi4py, which I installed using pip with intel compiler and intel MPI. I use the same compiler and MPI modules to run a batch job using Dask-MPI. I’m tryi… --- ## [Deploying Dask Gateway without Traefik](https://dask.discourse.group/t/deploying-dask-gateway-without-traefik/3554)
**Author:** [@Skyfree](https://dask.discourse.group/u/Skyfree)\ **Replies:** 1\ **Last updated:** [October 11, 2024, 3:18pm UTC](https://dask.discourse.group/t/deploying-dask-gateway-without-traefik/3554 "2024-10-11T15:18:41Z")
Hi folks, I am looking to deploy Dask Gateway via Helm in OCP4 in an enterprise setting and we have specific routing specifications and I am not able to install Traefik along with it’s CRDs. Is it possible to run dask … --- ## [Deploying Dask without cluster permissions (only with namespace permissions)](https://dask.discourse.group/t/deploying-dask-without-cluster-permissions-only-with-namespace-permissions/1605)
**Author:** [@VladimirVisnovsky](https://dask.discourse.group/u/VladimirVisnovsky)\ **Replies:** 11\ **Last updated:** [October 11, 2024, 12:47pm UTC](https://dask.discourse.group/t/deploying-dask-without-cluster-permissions-only-with-namespace-permissions/1605 "2024-10-11T12:47:07Z")
I am trying to deploy Dask with only Namespace permissions (without permissions to the whole cluster, meaning ClusterRole would be interpreted as Role). However, Gateway needs permissions to the whole cluster to work wit… --- ## [Deploying Dask on k8s without CRD's](https://dask.discourse.group/t/deploying-dask-on-k8s-without-crds/3186)
**Author:** [@dharhas](https://dask.discourse.group/u/dharhas)\ **Replies:** 4\ **Last updated:** [September 9, 2024, 9:11am UTC](https://dask.discourse.group/t/deploying-dask-on-k8s-without-crds/3186 "2024-09-09T09:11:20Z")
Is there a way to deploy dask on an existing k8s cluster where we only have access to a namespace but no ability to create CRD’s. Both dask-gateway and dask-kubernetes helm charts looks like they require the ability to c… --- ## [Creating a dask SSHCluster with a tunnel](https://dask.discourse.group/t/creating-a-dask-sshcluster-with-a-tunnel/2989)
**Author:** [@tusca99](https://dask.discourse.group/u/tusca99)\ **Replies:** 4\ **Last updated:** [September 3, 2024, 3:04pm UTC](https://dask.discourse.group/t/creating-a-dask-sshcluster-with-a-tunnel/2989 "2024-09-03T15:04:20Z")
Hello, I am trying to create a Dask cluster via SSH. I have 3 VMs under a subnet where I can access to a gate. I read dask and then asyncssh.connect documentation and discovered the keyword “tunnel” that can be passed … --- ## [Difference when starting a SLURM cluster with Conda from SLURM job or from Terminal](https://dask.discourse.group/t/difference-when-starting-a-slurm-cluster-with-conda-from-slurm-job-or-from-terminal/2967)
**Author:** [@RetoKrummenacher](https://dask.discourse.group/u/RetoKrummenacher)\ **Replies:** 3\ **Last updated:** [August 18, 2024, 1:00pm UTC](https://dask.discourse.group/t/difference-when-starting-a-slurm-cluster-with-conda-from-slurm-job-or-from-terminal/2967 "2024-08-18T13:00:17Z")
Hi everyone. First, thank you for creating Dask. As it seems very suitable to distribute work on multiple cluster nodes, I wanted to use it as part of my pipeline in a Conda environment to compute the embarrassingly par… --- ## [Dask on k8s - Best computing practices](https://dask.discourse.group/t/dask-on-k8s-best-computing-practices/2914)
**Author:** [@Lolomgrofl](https://dask.discourse.group/u/Lolomgrofl)\ **Replies:** 5\ **Last updated:** [July 25, 2024, 8:19am UTC](https://dask.discourse.group/t/dask-on-k8s-best-computing-practices/2914 "2024-07-25T08:19:20Z")
Hi everyone, I am planning to set up a Kubernetes cluster with 4 nodes (4 VMs) and deploy Dask on top of it. I am curious about the best practices for resource management at the pod (worker) level. Specifically, I want … --- ## [Error on invoking client in Azure Kubernetes Service cluster](https://dask.discourse.group/t/error-on-invoking-client-in-azure-kubernetes-service-cluster/2883)
**Author:** [@renjthmails](https://dask.discourse.group/u/renjthmails)\ **Replies:** 1\ **Last updated:** [July 11, 2024, 7:36am UTC](https://dask.discourse.group/t/error-on-invoking-client-in-azure-kubernetes-service-cluster/2883 "2024-07-11T07:36:44Z")
Hi Team, I am getting the following error on invoking submit using client Object. Any help will be highly appreciated. Also adding the scheduler side log at the bottom. root@python:~# python3 Python 3.10.12 (main, Aug… --- ## [Worker pods exist but client cannot connect to them or workers do not accept jobs](https://dask.discourse.group/t/worker-pods-exist-but-client-cannot-connect-to-them-or-workers-do-not-accept-jobs/2822)
**Author:** [@Alputer](https://dask.discourse.group/u/Alputer)\ **Replies:** 7\ **Last updated:** [June 27, 2024, 1:21pm UTC](https://dask.discourse.group/t/worker-pods-exist-but-client-cannot-connect-to-them-or-workers-do-not-accept-jobs/2822 "2024-06-27T13:21:50Z")
I have been trying to configure “Dask-Gateway” for a few days but am still not able to run any analysis from the client side in my trials. I have tried so many configurations, everything seems to be working fine, but job… --- ## [Unable to start DASK cluster in Kubeflow](https://dask.discourse.group/t/unable-to-start-dask-cluster-in-kubeflow/2800)
**Author:** [@Joish](https://dask.discourse.group/u/Joish)\ **Replies:** 4\ **Last updated:** [June 14, 2024, 1:25pm UTC](https://dask.discourse.group/t/unable-to-start-dask-cluster-in-kubeflow/2800 "2024-06-14T13:25:03Z")
I’m encountering an issue while trying to start a DASK cluster in Kubeflow. Here’s what I’ve done so far: I installed the DASK Kubernetes operator using the following Helm command: $ helm install --create-namespace -… --- ## [Dask deployment on SLURM Cluster with GPUs](https://dask.discourse.group/t/dask-deployment-on-slurm-cluster-with-gpus/2735)
**Author:** [@ZevVladimir](https://dask.discourse.group/u/ZevVladimir)\ **Replies:** 7\ **Last updated:** [May 20, 2024, 10:07am UTC](https://dask.discourse.group/t/dask-deployment-on-slurm-cluster-with-gpus/2735 "2024-05-20T10:07:29Z")
I am trying to get XGBoost with dask setup on a SLURM cluster and was able to get this setup with using only CPUs. However, I want to use GPUs to accelerate the training. To start with I did just a basic calculation whic… --- ## [Specifying default Conda env for Gateway containers](https://dask.discourse.group/t/specifying-default-conda-env-for-gateway-containers/2647)
**Author:** [@NickCote](https://dask.discourse.group/u/NickCote)\ **Replies:** 2\ **Last updated:** [April 19, 2024, 9:12pm UTC](https://dask.discourse.group/t/specifying-default-conda-env-for-gateway-containers/2647 "2024-04-19T21:12:32Z")
We run a JupyterHub deployed on Kubernetes with the DaskHub Helm chart. It uses a custom Jupyter image that contains a few different conda environments. This image is used for Dask Gateway as well via : DASK\_GATEWAY\_\_CLU… --- ## [Deploying Dask on an rke2 custom cluster](https://dask.discourse.group/t/deploying-dask-on-an-rke2-custom-cluster/2639)
**Author:** [@Shepard33](https://dask.discourse.group/u/Shepard33)\ **Replies:** 8\ **Last updated:** [April 4, 2024, 1:43pm UTC](https://dask.discourse.group/t/deploying-dask-on-an-rke2-custom-cluster/2639 "2024-04-04T13:43:29Z")
Hello I try, during several months now, to install Dask on a K8s cluster. I first try on an Openstack managed cluster Magnum but my kubernetes version was to old and i cannot install the Dask-gateway #Last helm chart … --- ## [Dask-jobqueue and SLURMCluster options](https://dask.discourse.group/t/dask-jobqueue-and-slurmcluster-options/2643)
**Author:** [@flipmostyle](https://dask.discourse.group/u/flipmostyle)\ **Replies:** 1\ **Last updated:** [March 26, 2024, 10:50am UTC](https://dask.discourse.group/t/dask-jobqueue-and-slurmcluster-options/2643 "2024-03-26T10:50:18Z")
This is my first foray into the Dask world and I’ve hit a stumbling block when setting up my Cluster options. If I don’t specify the memory required I get the following error: ValueError: You must specify how much cores… --- ## [Customize Autoscaling using dask's kubernetes operator](https://dask.discourse.group/t/customize-autoscaling-using-dasks-kubernetes-operator/2593)
**Author:** [@sil-lnagel](https://dask.discourse.group/u/sil-lnagel)\ **Replies:** 3\ **Last updated:** [March 14, 2024, 9:09am UTC](https://dask.discourse.group/t/customize-autoscaling-using-dasks-kubernetes-operator/2593 "2024-03-14T09:09:20Z")
Hi, we are currently trying to migrate from a static dask deployment with n workers to an adaptive deployment. For this we are trying to use the new dask kubernetes operator. When I tried to run some example workload w… --- ## [Gateway profiles in kubernetes](https://dask.discourse.group/t/gateway-profiles-in-kubernetes/2562)
**Author:** [@melaz](https://dask.discourse.group/u/melaz)\ **Replies:** 1\ **Last updated:** [February 28, 2024, 2:54pm UTC](https://dask.discourse.group/t/gateway-profiles-in-kubernetes/2562 "2024-02-28T14:54:11Z")
Hello community, I’ve installed dask-gateway in cluster and can run my example code into it, it perfectly spins workers, do math and shutsdown. Now my task is to make “profiles” so different teams get different worker im… --- ## [Daskhub Helm chart leads to client timeouts](https://dask.discourse.group/t/daskhub-helm-chart-leads-to-client-timeouts/2430)
**Author:** [@NickCote](https://dask.discourse.group/u/NickCote)\ **Replies:** 8\ **Last updated:** [February 27, 2024, 6:10pm UTC](https://dask.discourse.group/t/daskhub-helm-chart-leads-to-client-timeouts/2430 "2024-02-27T18:10:27Z")
Hello, We’ve been battling these timeout issues for quite some time and could use some guidance. In our troubleshooting efforts we’ve scaled back to a fresh kubernetes install, no ingress controllers, just bare vanilla … [Next page](https://dask.discourse.group/c/deploying-dask/9.md?page=1)