# Distributed **URL:** https://dask.discourse.group/c/distributed/10.md?page=2 [Latest](https://dask.discourse.group/latest.md) · [Categories](https://dask.discourse.group/categories.md) · [Tags](https://dask.discourse.group/tags.md) **Page:** 3 --- ## [Workers not scaling up despite tasks being locked by limited resource](https://dask.discourse.group/t/workers-not-scaling-up-despite-tasks-being-locked-by-limited-resource/3767)
**Author:** [@Fogapod](https://dask.discourse.group/u/Fogapod)\ **Replies:** 1\ **Last updated:** [January 15, 2025, 7:33am UTC](https://dask.discourse.group/t/workers-not-scaling-up-despite-tasks-being-locked-by-limited-resource/3767 "2025-01-15T07:33:58Z")
Dask and distributed versions: 2024.11.2 My autoscaler: apiVersion: kubernetes.dask.org/v1 kind: DaskAutoscaler metadata: name: dask-primary spec: cluster: dask-primary minimum: 1 # tried 2 as well maximum: 5 I… --- ## [FutureCancelledError: scheduler-connection-lost due to high load?](https://dask.discourse.group/t/futurecancellederror-scheduler-connection-lost-due-to-high-load/3742)
**Author:** [@bhofner](https://dask.discourse.group/u/bhofner)\ **Replies:** 8\ **Last updated:** [December 19, 2024, 8:46am UTC](https://dask.discourse.group/t/futurecancellederror-scheduler-connection-lost-due-to-high-load/3742 "2024-12-19T08:46:38Z")
Hello, im using Dask Distributed Version 2024.12.0 to distribute work on a single Linux machine to multiple processes. The machine has 8 cores. The worker function is kind of long running (extracting text from large P… --- ## [Reading data from netcdf or zarr files loads all data into memory](https://dask.discourse.group/t/reading-data-from-netcdf-or-zarr-files-loads-all-data-into-memory/3733)
**Author:** [@schlunma](https://dask.discourse.group/u/schlunma)\ **Replies:** 4\ **Last updated:** [December 18, 2024, 4:02pm UTC](https://dask.discourse.group/t/reading-data-from-netcdf-or-zarr-files-loads-all-data-into-memory/3733 "2024-12-18T16:02:32Z")
Hello! I am working with climate model output (mostly in netcdf format) and came across some issues when reading those data and calculating averages with a distributed scheduler. Looking at the Dask dashboard, I find th… --- ## [Multi-Threading on workers in Dask Distrubed (\>2024.3.0)](https://dask.discourse.group/t/multi-threading-on-workers-in-dask-distrubed-2024-3-0/3738)
**Author:** [@Lucas](https://dask.discourse.group/u/Lucas)\ **Replies:** 1\ **Last updated:** [December 12, 2024, 4:12pm UTC](https://dask.discourse.group/t/multi-threading-on-workers-in-dask-distrubed-2024-3-0/3738 "2024-12-12T16:12:04Z")
Hi, In previous versions of Dask (pre dask-expr integration), it was possible to run submit “multi-threaded” jobs in the following way: with ThreadPoolExecutor( max\_workers=num\_threads ) as ex: futu… --- ## [How to use Built-In WorkerPlugin to import code when worker spawns](https://dask.discourse.group/t/how-to-use-built-in-workerplugin-to-import-code-when-worker-spawns/3695)
**Author:** [@ofk123](https://dask.discourse.group/u/ofk123)\ **Replies:** 1\ **Last updated:** [November 27, 2024, 9:42am UTC](https://dask.discourse.group/t/how-to-use-built-in-workerplugin-to-import-code-when-worker-spawns/3695 "2024-11-27T09:42:35Z")
@martindurant or someone else. Can I ask a follow-up to your answer in Question about how to best store self-written functions, when using dask\_gateway - Pangeo Cloud Support - Pangeo ? When using cluster.adapt(), my q… --- ## [Splitting big NetCDF file into hundreds of smaller files](https://dask.discourse.group/t/splitting-big-netcdf-file-into-hundreds-of-smaller-files/3683)
**Author:** [@dogukanteber](https://dask.discourse.group/u/dogukanteber)\ **Replies:** 1\ **Last updated:** [November 22, 2024, 12:21pm UTC](https://dask.discourse.group/t/splitting-big-netcdf-file-into-hundreds-of-smaller-files/3683 "2024-11-22T12:21:34Z")
Hi all, I have 12 NetCDF files with total 190 GB in size. Some of these files are too big so I decided to split them by Y-axis before feeding them into our model. In our current implementation, we split these big files … --- ## [Dask-distributed RDataFrame on a SlurmCluster](https://dask.discourse.group/t/dask-distributed-rdataframe-on-a-slurmcluster/3642)
**Author:** [@Jindrich](https://dask.discourse.group/u/Jindrich)\ **Replies:** 3\ **Last updated:** [November 15, 2024, 8:46pm UTC](https://dask.discourse.group/t/dask-distributed-rdataframe-on-a-slurmcluster/3642 "2024-11-15T20:46:07Z")
Dear experts, When we were working with the ROOT framework, we attempted to use a distributed ROOT.RDF.Experimental.Distributed.Dask.RDataFrame on a Slurm cluster. We encountered some issues with this. Initially, this… --- ## [How to use client.compute() with sync=True](https://dask.discourse.group/t/how-to-use-client-compute-with-sync-true/3663)
**Author:** [@Tianming\_Han](https://dask.discourse.group/u/Tianming_Han)\ **Replies:** 1\ **Last updated:** [November 15, 2024, 8:13pm UTC](https://dask.discourse.group/t/how-to-use-client-compute-with-sync-true/3663 "2024-11-15T20:13:07Z")
I’m doing local asynchronous testing before setting up the code to be run on cluster. With client = Client(..., asynchronous=True) computing tasks need to be done with client.compute() which has a sync parameter. However… --- ## [Workers do not keep data from \`client.scatter(..., broadcast=True)\`](https://dask.discourse.group/t/workers-do-not-keep-data-from-client-scatter-broadcast-true/3656)
**Author:** [@Dahn](https://dask.discourse.group/u/Dahn)\ **Replies:** 1\ **Last updated:** [November 15, 2024, 4:40pm UTC](https://dask.discourse.group/t/workers-do-not-keep-data-from-client-scatter-broadcast-true/3656 "2024-11-15T16:40:25Z")
When using client.scatter(..., broadcast=True), the data gets briefly scattered to all workers but then is immediately removed from all but one. My expectation would be that all workers retain the data. Happy to report … --- ## [Xarray operations (e.g., preprocess) running locally (post open\_mfdataset) instead of on Dask distributed cluster](https://dask.discourse.group/t/xarray-operations-e-g-preprocess-running-locally-post-open-mfdataset-instead-of-on-dask-distributed-cluster/3622)
**Author:** [@lbesnard](https://dask.discourse.group/u/lbesnard)\ **Replies:** 7\ **Last updated:** [November 15, 2024, 3:36pm UTC](https://dask.discourse.group/t/xarray-operations-e-g-preprocess-running-locally-post-open-mfdataset-instead-of-on-dask-distributed-cluster/3622 "2024-11-15T15:36:36Z")
FYI, I’ve also posted the same question at Pangeo and Coiled. ( I’m a new user so only limited to 2 links per post). I will update this discussion in case I have a positive answer from those places. I’m using dask.distr… --- ## [How to parallelize several loops on huge climate datasets using dask.delayed](https://dask.discourse.group/t/how-to-parallelize-several-loops-on-huge-climate-datasets-using-dask-delayed/3627)
**Author:** [@Arty](https://dask.discourse.group/u/Arty)\ **Replies:** 7\ **Last updated:** [November 8, 2024, 3:36pm UTC](https://dask.discourse.group/t/how-to-parallelize-several-loops-on-huge-climate-datasets-using-dask-delayed/3627 "2024-11-08T15:36:25Z")
Hello, I’ve been struggling on that matter for more than a week now and have tried so many things that I finally abdicated and came here to ask for your help. Context: I’ve got outputs from several WRF model configurat… --- ## [Dask.distributed configurations benchmark](https://dask.discourse.group/t/dask-distributed-configurations-benchmark/3609)
**Author:** [@Arty](https://dask.discourse.group/u/Arty)\ **Replies:** 2\ **Last updated:** [November 3, 2024, 4:53pm UTC](https://dask.discourse.group/t/dask-distributed-configurations-benchmark/3609 "2024-11-03T16:53:05Z")
Hello, I would like to right a short benchmark script to test for various Dask.distributed Client configurations, but I don’t get the expected results as cache and memory data do not seem to be cleared properly. I’d li… --- ## [Install dependencies on EC2Cluster](https://dask.discourse.group/t/install-dependencies-on-ec2cluster/3579)
**Author:** [@Sacha\_Van\_Weeren](https://dask.discourse.group/u/Sacha_Van_Weeren)\ **Replies:** 3\ **Last updated:** [October 24, 2024, 11:45am UTC](https://dask.discourse.group/t/install-dependencies-on-ec2cluster/3579 "2024-10-24T11:45:48Z")
When start an EC2Cluster with cluster = EC2Cluster(region="us-east-1", # the Name is the IAM role name iam\_instance\_profile= {'Name': 'dask-cluster-ec2-… --- ## [Online Data Generation and Streaming for ML Applications](https://dask.discourse.group/t/online-data-generation-and-streaming-for-ml-applications/3233)
**Author:** [@LTMeyer](https://dask.discourse.group/u/LTMeyer)\ **Replies:** 5\ **Last updated:** [October 23, 2024, 5:18pm UTC](https://dask.discourse.group/t/online-data-generation-and-streaming-for-ml-applications/3233 "2024-10-23T17:18:17Z")
Context I’m working on ML applications for Science where we want to generate data from numerical simulations that are typically MPI-based programs running on multiple cores. As we are running on a HPC cluster for which n… --- ## [WorkerPlugin in Airflow: No module named 'unusual\_prefix\_\*'](https://dask.discourse.group/t/workerplugin-in-airflow-no-module-named-unusual-prefix/3583)
**Author:** [@ChristianRothQC](https://dask.discourse.group/u/ChristianRothQC)\ **Replies:** 2\ **Last updated:** [October 19, 2024, 1:41pm UTC](https://dask.discourse.group/t/workerplugin-in-airflow-no-module-named-unusual-prefix/3583 "2024-10-19T13:41:41Z")
Observation I am currently experimenting with vertical scaling with dask from inside an airflow task. Using the WorkerPlugin seems to interfere with the orchestration framework module hacking magic for me: AIRFLOW\_\_COR… --- ## [Common workflow for using Dask on HPC systems](https://dask.discourse.group/t/common-workflow-for-using-dask-on-hpc-systems/3573)
**Author:** [@joseph-pq](https://dask.discourse.group/u/joseph-pq)\ **Replies:** 2\ **Last updated:** [October 18, 2024, 4:32pm UTC](https://dask.discourse.group/t/common-workflow-for-using-dask-on-hpc-systems/3573 "2024-10-18T16:32:21Z")
Hi! I’m new to Dask. I’m currently working in an HPC managed by SLURM with some compute nodes (those that execute the jobs) and the login node (which I access through SSH to send the SLURM jobs). I’m looking to define my… --- ## [Dask Controller (Dask Gateway) Sometimes Hanges](https://dask.discourse.group/t/dask-controller-dask-gateway-sometimes-hanges/3020)
**Author:** [@jbeeman](https://dask.discourse.group/u/jbeeman)\ **Replies:** 5\ **Last updated:** [October 13, 2024, 2:40am UTC](https://dask.discourse.group/t/dask-controller-dask-gateway-sometimes-hanges/3020 "2024-10-13T02:40:24Z")
Hello All! We’ve been running into a problem that shows up randomly with our dask gateway cluster. Sometimes out of nowhere, the dask gateway controller will stop receiving requests to create the cluster. The API says i… --- ## [Merging hundreds of NetCDF files into a single big NetCDF file on HPC Cluster](https://dask.discourse.group/t/merging-hundreds-of-netcdf-files-into-a-single-big-netcdf-file-on-hpc-cluster/3447)
**Author:** [@dogukanteber](https://dask.discourse.group/u/dogukanteber)\ **Replies:** 8\ **Last updated:** [October 4, 2024, 8:48pm UTC](https://dask.discourse.group/t/merging-hundreds-of-netcdf-files-into-a-single-big-netcdf-file-on-hpc-cluster/3447 "2024-10-04T20:48:35Z")
Hi all, We have a small-scaled HPC cluster on GCP. We use Slurm as a job queue and Lustre as a parallel file system. Since the data we work with is too big, we first split the input data into smaller batches and run the… --- ## [Runtime of currently processing jobs to cancel/restart long running jobs](https://dask.discourse.group/t/runtime-of-currently-processing-jobs-to-cancel-restart-long-running-jobs/3534)
**Author:** [@pculviner](https://dask.discourse.group/u/pculviner)\ **Replies:** 2\ **Last updated:** [October 4, 2024, 8:01pm UTC](https://dask.discourse.group/t/runtime-of-currently-processing-jobs-to-cancel-restart-long-running-jobs/3534 "2024-10-04T20:01:59Z")
Hi all, Apologies if this has been answered before and I couldn’t find the search term. I using job\_queue to run about 10-20 thousand fairly long running embarrassingly parallel jobs (5-10 minutes) as futures submitted … --- ## [Trying to shutdown workers with completed tasks in order to reduce costs](https://dask.discourse.group/t/trying-to-shutdown-workers-with-completed-tasks-in-order-to-reduce-costs/3523)
**Author:** [@AlverGant](https://dask.discourse.group/u/AlverGant)\ **Replies:** 7\ **Last updated:** [October 4, 2024, 7:47pm UTC](https://dask.discourse.group/t/trying-to-shutdown-workers-with-completed-tasks-in-order-to-reduce-costs/3523 "2024-10-04T19:47:05Z")
Hello, I’m encountering a problem where Dask Distributed workers are not being properly shut down after completing their tasks. Despite successfully processing the tasks, the Dask scheduler keeps waiting for task comple… --- ## [Limit number of queued tasks per worker](https://dask.discourse.group/t/limit-number-of-queued-tasks-per-worker/3026)
**Author:** [@fraimondo](https://dask.discourse.group/u/fraimondo)\ **Replies:** 3\ **Last updated:** [October 4, 2024, 4:39pm UTC](https://dask.discourse.group/t/limit-number-of-queued-tasks-per-worker/3026 "2024-10-04T16:39:04Z")
I am struggling with this odd situation for quite some time. I hope I can get some help around. My issue is quite simple: I have N long-running tasks (12~24 hours each) which I want to parallelize using dask distributed… --- ## [Dask Cluster on k8s - Managing Multiple Users Submitting Jobs Concurrently](https://dask.discourse.group/t/dask-cluster-on-k8s-managing-multiple-users-submitting-jobs-concurrently/3492)
**Author:** [@Lolomgrofl](https://dask.discourse.group/u/Lolomgrofl)\ **Replies:** 8\ **Last updated:** [October 4, 2024, 3:57pm UTC](https://dask.discourse.group/t/dask-cluster-on-k8s-managing-multiple-users-submitting-jobs-concurrently/3492 "2024-10-04T15:57:16Z")
Hi everyone. We’ve deployed Dask cluster on top of k8s ( 5 nodes, each having 200GB of RAM and 50 vCPUs), using an example for this link. When a single developer is using the cluster, everything works perfectly. However… --- ## [How do task priorities work?](https://dask.discourse.group/t/how-do-task-priorities-work/2878)
**Author:** [@byrom771](https://dask.discourse.group/u/byrom771)\ **Replies:** 11\ **Last updated:** [October 4, 2024, 3:52pm UTC](https://dask.discourse.group/t/how-do-task-priorities-work/2878 "2024-10-04T15:52:02Z")
As far as I know, when calling client.submit(…,priority=-10), that task has lower priority than when calling client.submit(…,priority=10). However, when browsing through the workers in the Dask dashboard (attached image… --- ## [Dask with Azure ML Studio: How can I scale Distributed horizontally in a Notebook?](https://dask.discourse.group/t/dask-with-azure-ml-studio-how-can-i-scale-distributed-horizontally-in-a-notebook/2854)
**Author:** [@rsignell](https://dask.discourse.group/u/rsignell)\ **Replies:** 4\ **Last updated:** [September 30, 2024, 12:57pm UTC](https://dask.discourse.group/t/dask-with-azure-ml-studio-how-can-i-scale-distributed-horizontally-in-a-notebook/2854 "2024-09-30T12:57:09Z")
Using Azure ML Studio, you can fire up JupyterLab from a compute instance, and you can start an ML cluster. But does anyone have a good way to work with a Distributed cluster scaling horizontally in this environment? … --- ## [Dilemma: Schedule IO-Bound / CPU-Bound tasks in cascaded clients](https://dask.discourse.group/t/dilemma-schedule-io-bound-cpu-bound-tasks-in-cascaded-clients/3004)
**Author:** [@bozden](https://dask.discourse.group/u/bozden)\ **Replies:** 11\ **Last updated:** [September 27, 2024, 6:17pm UTC](https://dask.discourse.group/t/dilemma-schedule-io-bound-cpu-bound-tasks-in-cascaded-clients/3004 "2024-09-27T18:17:24Z")
First of all, thank you for this excellent project - from idea to documentation. BG: I’m coding a CLI application about Voice-AI datasets (different flavors, versions, languages) where a user can specify what to do on w… --- ## [HTCondorCluster failed to start - ConnectionRefusedError: \[Errno 111\] Connection refused](https://dask.discourse.group/t/htcondorcluster-failed-to-start-connectionrefusederror-errno-111-connection-refused/3502)
**Author:** [@Earl\_Russell\_Almazan](https://dask.discourse.group/u/Earl_Russell_Almazan)\ **Replies:** 1\ **Last updated:** [September 27, 2024, 1:50pm UTC](https://dask.discourse.group/t/htcondorcluster-failed-to-start-connectionrefusederror-errno-111-connection-refused/3502 "2024-09-27T13:50:30Z")
Running on UChicago and was waiting for compute() with an HTCondorCluster running: cluster = HTCondorCluster(log\_directory="path/to/log/", cores=5, memory="20GB", disk="5GB") output = \[ \] for i in loop: output.appe… --- ## [How do I modularize functions that work with dask?](https://dask.discourse.group/t/how-do-i-modularize-functions-that-work-with-dask/3224)
**Author:** [@2dask-crls](https://dask.discourse.group/u/2dask-crls)\ **Replies:** 7\ **Last updated:** [September 20, 2024, 4:16pm UTC](https://dask.discourse.group/t/how-do-i-modularize-functions-that-work-with-dask/3224 "2024-09-20T16:16:45Z")
I’m trying to modularize my functions that use Dask, but I keep encountering the error "No module named 'my\_module'". I can’t import any local module that is related to Dask, and currently, everything I do with Dask has … --- ## [Dask-mpi and mpirun - Can't scale up run with larger files and or more core/node on HPC cluster](https://dask.discourse.group/t/dask-mpi-and-mpirun-cant-scale-up-run-with-larger-files-and-or-more-core-node-on-hpc-cluster/3415)
**Author:** [@Antje](https://dask.discourse.group/u/Antje)\ **Replies:** 2\ **Last updated:** [September 20, 2024, 4:03pm UTC](https://dask.discourse.group/t/dask-mpi-and-mpirun-cant-scale-up-run-with-larger-files-and-or-more-core-node-on-hpc-cluster/3415 "2024-09-20T16:03:28Z")
I am trying to implement a distributed run over several nodes and cores on an HPC cluster using the slurm submission system. I successfully tested the implementation of my code in an interactive session and it runs fine.… --- ## [Distributed Scheduler becoming the bottleneck](https://dask.discourse.group/t/distributed-scheduler-becoming-the-bottleneck/3371)
**Author:** [@RaphaelRobidas](https://dask.discourse.group/u/RaphaelRobidas)\ **Replies:** 5\ **Last updated:** [September 17, 2024, 6:32pm UTC](https://dask.discourse.group/t/distributed-scheduler-becoming-the-bottleneck/3371 "2024-09-17T18:32:07Z")
I am running a Dask cluster on HPC resources using distributed and dask-jobqueue. I separated the scheduler into its own SLURM job, and the clients connect to the address of the scheduler. Workers are requested to SLURM … --- ## [How does dask schedule to (logical-)cores?](https://dask.discourse.group/t/how-does-dask-schedule-to-logical-cores/3146)
**Author:** [@bozden](https://dask.discourse.group/u/bozden)\ **Replies:** 8\ **Last updated:** [September 13, 2024, 2:49pm UTC](https://dask.discourse.group/t/how-does-dask-schedule-to-logical-cores/3146 "2024-09-13T14:49:56Z")
I’m feeding dask with chucks of data to process (details here). I’m using all logical cores as sub-processes also include read/write operations (mixed load). I also specify load type in resources of the LocalCluster: c… [Previous page](https://dask.discourse.group/c/distributed/10.md?page=1) [Next page](https://dask.discourse.group/c/distributed/10.md?page=3)