# Distributed **URL:** https://dask.discourse.group/c/distributed/10.md?page=1 [Latest](https://dask.discourse.group/latest.md) · [Categories](https://dask.discourse.group/categories.md) · [Tags](https://dask.discourse.group/tags.md) **Page:** 2 --- ## [Iterative array update using persist() - Scheduler tasks accumulation](https://dask.discourse.group/t/iterative-array-update-using-persist-scheduler-tasks-accumulation/4061)
**Author:** [@robin-cls](https://dask.discourse.group/u/robin-cls)\ **Replies:** 3\ **Last updated:** [August 29, 2025, 12:48pm UTC](https://dask.discourse.group/t/iterative-array-update-using-persist-scheduler-tasks-accumulation/4061 "2025-08-29T12:48:29Z")
mercredi 20 août 2025 15:25 Hello, I am working on a case where I want to update an array located on the cluster. In the little example below, this array fits in memory but in real life, it will be much bigger. Thus, … --- ## [\[Help\] How can I get my scheduler to not think it is Jan 1st 1970?](https://dask.discourse.group/t/help-how-can-i-get-my-scheduler-to-not-think-it-is-jan-1st-1970/4064)
**Author:** [@Joseph-Rubin\_brevan](https://dask.discourse.group/u/Joseph-Rubin_brevan)\ **Replies:** 8\ **Last updated:** [August 28, 2025, 10:02pm UTC](https://dask.discourse.group/t/help-how-can-i-get-my-scheduler-to-not-think-it-is-jan-1st-1970/4064 "2025-08-28T22:02:37Z")
Hi, when I deploy my cluster (on AWS fargate) the distributed cluster seems to think it is the Unix Epoch instead of the real time. How can I fix this? It actually started a little bit before the Epoch. I did a manual … --- ## [Configuring Dask to target GPU or CPU workers in a heterogeneous cluster](https://dask.discourse.group/t/configuring-dask-to-target-gpu-or-cpu-workers-in-a-heterogeneous-cluster/4046)
**Author:** [@doughng](https://dask.discourse.group/u/doughng)\ **Replies:** 1\ **Last updated:** [August 6, 2025, 1:03pm UTC](https://dask.discourse.group/t/configuring-dask-to-target-gpu-or-cpu-workers-in-a-heterogeneous-cluster/4046 "2025-08-06T13:03:06Z")
I’m working with a manually deployed Dask cluster across multiple machines. Some of the worker nodes have GPUs, while others do not. I’d like to understand how to group or label Dask workers such that: Workloads that… --- ## [Let jobs finish when using adapt interface with jobqueues with slurm](https://dask.discourse.group/t/let-jobs-finish-when-using-adapt-interface-with-jobqueues-with-slurm/4020)
**Author:** [@M1Sports20](https://dask.discourse.group/u/M1Sports20)\ **Replies:** 4\ **Last updated:** [July 18, 2025, 5:33pm UTC](https://dask.discourse.group/t/let-jobs-finish-when-using-adapt-interface-with-jobqueues-with-slurm/4020 "2025-07-18T17:33:37Z")
Hello. I am using the concurrent futures interface with dask. I have long running tasks that are running in dask ontop of slurm. I am using the adapt interface to scale my workload. The one thing I don’t like is that … --- ## [Optimal way to monitor GPU memory usage during distributed training (XGBoost)](https://dask.discourse.group/t/optimal-way-to-monitor-gpu-memory-usage-during-distributed-training-xgboost/3998)
**Author:** [@ap213](https://dask.discourse.group/u/ap213)\ **Replies:** 4\ **Last updated:** [July 18, 2025, 12:37pm UTC](https://dask.discourse.group/t/optimal-way-to-monitor-gpu-memory-usage-during-distributed-training-xgboost/3998 "2025-07-18T12:37:38Z")
GPU Memory Monitoring During Distributed XGBoost Training Hello, I was wondering if anyone had any advice or information they could direct me to regarding ways to monitor GPU memory usage during distributed training us… --- ## [I recently built a project that parallelizes multiple transformers on a single GPU using Dask, would love to hear your feedback](https://dask.discourse.group/t/i-recently-built-a-project-that-parallelizes-multiple-transformers-on-a-single-gpu-using-dask-would-love-to-hear-your-feedback/4025)
**Author:** [@Rishikesh\_gharat](https://dask.discourse.group/u/Rishikesh_gharat)\ **Replies:** 0\ **Last updated:** [July 17, 2025, 4:23pm UTC](https://dask.discourse.group/t/i-recently-built-a-project-that-parallelizes-multiple-transformers-on-a-single-gpu-using-dask-would-love-to-hear-your-feedback/4025 "2025-07-17T16:23:54Z")
this project was faster than using minikube containers, and gave me a great throughput of 400k reviews in 3 minutes. the projects aims to first categorize and then, summarize, these categorized reviews. the core of thi… --- ## [Concurrent futures, slurm, and adapt](https://dask.discourse.group/t/concurrent-futures-slurm-and-adapt/4019)
**Author:** [@M1Sports20](https://dask.discourse.group/u/M1Sports20)\ **Replies:** 2\ **Last updated:** [July 11, 2025, 7:26pm UTC](https://dask.discourse.group/t/concurrent-futures-slurm-and-adapt/4019 "2025-07-11T19:26:47Z")
I have a setup where a python script kicks of a slurm cluster of dasks tasks. It is using the concurrent futures interface. I am using the interface adapt(minimum=1, maximum=500). However, it never scales up past 1 da… --- ## [Unexpected dask argument](https://dask.discourse.group/t/unexpected-dask-argument/3979)
**Author:** [@sar49](https://dask.discourse.group/u/sar49)\ **Replies:** 2\ **Last updated:** [June 20, 2025, 12:50pm UTC](https://dask.discourse.group/t/unexpected-dask-argument/3979 "2025-06-20T12:50:15Z")
I get the following error when having a dask command in a SLURM script. The following is the script section that is erroring and the corresponding error. I am not sure how to continue. dask worker --scheduler-file “$SCH… --- ## [Persistent memory profiling/logging](https://dask.discourse.group/t/persistent-memory-profiling-logging/3956)
**Author:** [@folterj](https://dask.discourse.group/u/folterj)\ **Replies:** 7\ **Last updated:** [June 16, 2025, 6:12am UTC](https://dask.discourse.group/t/persistent-memory-profiling-logging/3956 "2025-06-16T06:12:29Z")
Hi, as part of debugging large Dask image operations run on HPC, I’m looking for a way to do real-time logging of (cluster) memory. I’ve tried a number of options using the functionality provided by Dask: The Dask das… --- ## [Access / re-initialize futures from multiple clients](https://dask.discourse.group/t/access-re-initialize-futures-from-multiple-clients/3975)
**Author:** [@nickvazz](https://dask.discourse.group/u/nickvazz)\ **Replies:** 1\ **Last updated:** [June 13, 2025, 5:15pm UTC](https://dask.discourse.group/t/access-re-initialize-futures-from-multiple-clients/3975 "2025-06-13T17:15:46Z")
Hello, Background I recently upgraded from distributed==2023.7.0 to distributed==2025.5.1. Upon upgrading I found an interesting change that broke some of my workflows and looking to understand if there is a better way … --- ## [Troubleshooting intermittent hanging behavior with one worker stuck running](https://dask.discourse.group/t/troubleshooting-intermittent-hanging-behavior-with-one-worker-stuck-running/3852)
**Author:** [@Kyle\_Redilla](https://dask.discourse.group/u/Kyle_Redilla)\ **Replies:** 3\ **Last updated:** [June 13, 2025, 3:35pm UTC](https://dask.discourse.group/t/troubleshooting-intermittent-hanging-behavior-with-one-worker-stuck-running/3852 "2025-06-13T15:35:13Z")
I am using xclim to do some bias-adjustment of climate data (zarr). Things seems to be working until the very end, where processing seems to halt near completion. Below is what the dashboard looks like when it’s stuck. … --- ## [Best Practices for Running Dask Clients with Local Code on a Shared Remote Cluster](https://dask.discourse.group/t/best-practices-for-running-dask-clients-with-local-code-on-a-shared-remote-cluster/3968)
**Author:** [@doughng](https://dask.discourse.group/u/doughng)\ **Replies:** 1\ **Last updated:** [June 6, 2025, 2:54pm UTC](https://dask.discourse.group/t/best-practices-for-running-dask-clients-with-local-code-on-a-shared-remote-cluster/3968 "2025-06-06T14:54:53Z")
I’m trying to understand the correct way to set up a Dask cluster across remote machines and deploy multiple Dask clients to it, where each client has its own set of Python modules. For example, I have a project structu… --- ## [Tasks forgotten waiting for new workers to be allocated](https://dask.discourse.group/t/tasks-forgotten-waiting-for-new-workers-to-be-allocated/3916)
**Author:** [@josephmure](https://dask.discourse.group/u/josephmure)\ **Replies:** 8\ **Last updated:** [June 6, 2025, 2:51pm UTC](https://dask.discourse.group/t/tasks-forgotten-waiting-for-new-workers-to-be-allocated/3916 "2025-06-06T14:51:18Z")
Hello!Monitoring the Dask dashboard, we could see the first tasks being completed. The first 10 tasks wer @jschueller and I are new users of SLURMCLuster, and we are trying to get the hang of it on a toy use case. Apol… --- ## [Per-worker (i.e., process) numpy array](https://dask.discourse.group/t/per-worker-i-e-process-numpy-array/3967)
**Author:** [@ilan-gold](https://dask.discourse.group/u/ilan-gold)\ **Replies:** 1\ **Last updated:** [June 6, 2025, 8:22am UTC](https://dask.discourse.group/t/per-worker-i-e-process-numpy-array/3967 "2025-06-06T08:22:38Z")
Hello! Let’s say I create a new dask array via map\_blocks i.e., chunks on the fly. If I wish to pass a numpy array to every process only once to be reused in all chunks (or better yet, although I think not possible, to… --- ## [Dask-scratch-space](https://dask.discourse.group/t/dask-scratch-space/3961)
**Author:** [@liberabaci](https://dask.discourse.group/u/liberabaci)\ **Replies:** 1\ **Last updated:** [June 6, 2025, 8:19am UTC](https://dask.discourse.group/t/dask-scratch-space/3961 "2025-06-06T08:19:10Z")
I’m launching some large workflows and I’ve noticed that my .dask/dask-scratch-space is getting sizable. When is it same to remove these files? can one do so as soon as the workers are lost? What are the scenarios where … --- ## [LightGBM Distributed Training](https://dask.discourse.group/t/lightgbm-distributed-training/3837)
**Author:** [@Matteo\_Ciccozzi](https://dask.discourse.group/u/Matteo_Ciccozzi)\ **Replies:** 10\ **Last updated:** [May 29, 2025, 3:20pm UTC](https://dask.discourse.group/t/lightgbm-distributed-training/3837 "2025-05-29T15:20:55Z")
Hi, I am training a distributed lgbm model using Dask, I have parquet data and I load it using read\_parquet utility. I then split the df to extract the data and the regression target. So I end up with two different data… --- ## [Any way to group / name workers and tasks?](https://dask.discourse.group/t/any-way-to-group-name-workers-and-tasks/3469)
**Author:** [@bozden](https://dask.discourse.group/u/bozden)\ **Replies:** 6\ **Last updated:** [May 10, 2025, 4:01pm UTC](https://dask.discourse.group/t/any-way-to-group-name-workers-and-tasks/3469 "2025-05-10T16:01:53Z")
Here is the workflow: Create a LocalCluster with N+M workers Main thread creates N outer workers (futures) with keys like “ds=en”, “ds=de”,… (language datasets). This is OK, they are not very much, and I can see them i… --- ## [Nanny Forces Single Core Usage](https://dask.discourse.group/t/nanny-forces-single-core-usage/3928)
**Author:** [@mronda](https://dask.discourse.group/u/mronda)\ **Replies:** 1\ **Last updated:** [May 9, 2025, 3:20pm UTC](https://dask.discourse.group/t/nanny-forces-single-core-usage/3928 "2025-05-09T15:20:20Z")
Hi all, I am running into an odd issue whenever my tasks call a compiled C++ executable (via subprocess). With the Nanny=True (default), each task gets pinned to 1 CPU core (not what I want), but as soon as I start work… --- ## [ KilledWorker Errors with SLURMCluster \`adapt()\` but not \`scale()\`](https://dask.discourse.group/t/killedworker-errors-with-slurmcluster-adapt-but-not-scale/3919)
**Author:** [@Rama\_Krishna\_Reddy](https://dask.discourse.group/u/Rama_Krishna_Reddy)\ **Replies:** 1\ **Last updated:** [May 2, 2025, 11:03am UTC](https://dask.discourse.group/t/killedworker-errors-with-slurmcluster-adapt-but-not-scale/3919 "2025-05-02T11:03:09Z")
Hi Dask Community, I’m encountering KilledWorker errors when using adaptive scaling (cluster.adapt()) with dask\_jobqueue.SLURMCluster (via a wrapper), but the same setup works fine when using fixed scaling (cluster.scal… --- ## [Testing lazy evaluation of task graphs](https://dask.discourse.group/t/testing-lazy-evaluation-of-task-graphs/1401)
**Author:** [@luk](https://dask.discourse.group/u/luk)\ **Replies:** 3\ **Last updated:** [April 24, 2025, 3:23pm UTC](https://dask.discourse.group/t/testing-lazy-evaluation-of-task-graphs/1401 "2025-04-24T15:23:32Z")
Hi everyone! This is a question about testing the lazy evaluation of dask task graphs! We’re using Dask+Xarray for a library of processes that can be chained together to manipulate large amounts of Earth Observation da… --- ## [Clarification sought on local scheduler and remote worker set up though SSHCluster](https://dask.discourse.group/t/clarification-sought-on-local-scheduler-and-remote-worker-set-up-though-sshcluster/3899)
**Author:** [@liuzongyue6](https://dask.discourse.group/u/liuzongyue6)\ **Replies:** 1\ **Last updated:** [April 18, 2025, 9:12am UTC](https://dask.discourse.group/t/clarification-sought-on-local-scheduler-and-remote-worker-set-up-though-sshcluster/3899 "2025-04-18T09:12:29Z")
I am targeting to run local scheduler and remote cluster worker, they are connected through VPN, only port 22 is open Terminal 1: At local computer, for scheduler Starts the Dask scheduler on local machine listening on… --- ## [Best way to persist different datasets in scaling workers](https://dask.discourse.group/t/best-way-to-persist-different-datasets-in-scaling-workers/3878)
**Author:** [@edgar-s-silva-alb](https://dask.discourse.group/u/edgar-s-silva-alb)\ **Replies:** 3\ **Last updated:** [April 3, 2025, 4:49pm UTC](https://dask.discourse.group/t/best-way-to-persist-different-datasets-in-scaling-workers/3878 "2025-04-03T16:49:20Z")
I have an application that contain ‘zones’ each zone loads a big dataset (that sadly can’t be partitioned). Then each zone will fire hundres of thosands of tasks that each read this dataset and perform some operations an… --- ## [Bad performance while training model from SQL data using Dask cluster](https://dask.discourse.group/t/bad-performance-while-training-model-from-sql-data-using-dask-cluster/3859)
**Author:** [@Glebbot](https://dask.discourse.group/u/Glebbot)\ **Replies:** 2\ **Last updated:** [March 19, 2025, 4:06pm UTC](https://dask.discourse.group/t/bad-performance-while-training-model-from-sql-data-using-dask-cluster/3859 "2025-03-19T16:06:21Z")
Hello everyone I came across a problem when I was trying to train a model on data from PostgreSQL table using dask read\_sql\_table function. I have a 12 million rows dataset and sklearn pipeline that looks like this: i… --- ## [Using Pytest with LocalClusters](https://dask.discourse.group/t/using-pytest-with-localclusters/1053)
**Author:** [@PGijsbers](https://dask.discourse.group/u/PGijsbers)\ **Replies:** 3\ **Last updated:** [March 18, 2025, 3:04pm UTC](https://dask.discourse.group/t/using-pytest-with-localclusters/1053 "2025-03-18T15:04:16Z")
Hi, I have a set of unit tests which internally create and tear down local clusters: def test\_one(): with LocalCluster() as cluster: # do something def test\_two(): with LocalCluster() as cluster: # do some… --- ## [Random FutureCancelledError() with unknown cause inside computes](https://dask.discourse.group/t/random-futurecancellederror-with-unknown-cause-inside-computes/3793)
**Author:** [@Fogapod](https://dask.discourse.group/u/Fogapod)\ **Replies:** 6\ **Last updated:** [January 25, 2025, 12:13pm UTC](https://dask.discourse.group/t/random-futurecancellederror-with-unknown-cause-inside-computes/3793 "2025-01-25T12:13:13Z")
I added docling as a part of dask pipeline recently. Docling is a document processor with it’s own thread pool which might be causing some contention with dask. I limit it to 1/5th of available cores and in general aside… --- ## [Trouble with priorities](https://dask.discourse.group/t/trouble-with-priorities/3784)
**Author:** [@Neotroglodyte](https://dask.discourse.group/u/Neotroglodyte)\ **Replies:** 13\ **Last updated:** [February 17, 2025, 11:30am UTC](https://dask.discourse.group/t/trouble-with-priorities/3784 "2025-02-17T11:30:00Z")
Hey guys. I use dask and distributed packages to run automation in my project. To speed up test execution (the tests cannot be run on a single environment in parallel), I spin up multiple VMs (dask workers) and then cr… --- ## [Get\_worker in client.run: raise ValueError("No worker found") from None](https://dask.discourse.group/t/get-worker-in-client-run-raise-valueerror-no-worker-found-from-none/3826)
**Author:** [@ChristianRothQC](https://dask.discourse.group/u/ChristianRothQC)\ **Replies:** 1\ **Last updated:** [February 14, 2025, 7:35pm UTC](https://dask.discourse.group/t/get-worker-in-client-run-raise-valueerror-no-worker-found-from-none/3826 "2025-02-14T19:35:11Z")
This simple code gives errors out with Exception: ValueError('No worker found') for me. from distributed import Client, LocalCluster, get\_worker def return\_one(): worker = get\_worker() return 1 def main(): … --- ## [How does batch runner setup dask worker](https://dask.discourse.group/t/how-does-batch-runner-setup-dask-worker/3804)
**Author:** [@EtienneReboul](https://dask.discourse.group/u/EtienneReboul)\ **Replies:** 3\ **Last updated:** [February 7, 2025, 2:29pm UTC](https://dask.discourse.group/t/how-does-batch-runner-setup-dask-worker/3804 "2025-02-07T14:29:03Z")
Hello everyone, I am working on an HPC cluster and been using the batch runner of dask-jobqueue. I have to use it because ssh tunneling is disabled for supposedly security reason. Furthermore admins prefer large job w… --- ## [Cleaning Up Errored Futures](https://dask.discourse.group/t/cleaning-up-errored-futures/3794)
**Author:** [@joecus1](https://dask.discourse.group/u/joecus1)\ **Replies:** 2\ **Last updated:** [January 24, 2025, 3:54pm UTC](https://dask.discourse.group/t/cleaning-up-errored-futures/3794 "2025-01-24T15:54:01Z")
Hi, Does anyone here have advice on cleaning up errored futures in a Distributed dask cluster? We have a long running distributed cluster we basically use as a web backend. When tasks error, they are held for a long tim… --- ## [Advice on how to structure Dask computation](https://dask.discourse.group/t/advice-on-how-to-structure-dask-computation/3766)
**Author:** [@liberabaci](https://dask.discourse.group/u/liberabaci)\ **Replies:** 7\ **Last updated:** [January 16, 2025, 3:28pm UTC](https://dask.discourse.group/t/advice-on-how-to-structure-dask-computation/3766 "2025-01-16T15:28:05Z")
I’m running a computation that is shared between a local machine and a Slurm cluster. On the local machine I run: start a dask client to the SLURMCluster a for loop generates a set of data vectors of length N from a v… [Previous page](https://dask.discourse.group/c/distributed/10.md) [Next page](https://dask.discourse.group/c/distributed/10.md?page=2)