# Latest **URL:** https://dask.discourse.group/latest.md [Latest](https://dask.discourse.group/latest.md) · [Categories](https://dask.discourse.group/categories.md) · [Tags](https://dask.discourse.group/tags.md) --- ## [Welcome to the Dask community forum](https://dask.discourse.group/t/welcome-to-the-dask-community-forum/7)
**Author:** [@system](https://dask.discourse.group/u/system)\ **Replies:** 0\ **Last updated:** [October 21, 2021, 6:45am UTC](https://dask.discourse.group/t/welcome-to-the-dask-community-forum/7 "2021-10-21T06:45:48Z")
Welcome to the Dask community forum This is a place for the Dask community to get to know one another, ask questions, swap stories, and generally be helpful. If you are new here, please read this orientation topic, and f… --- ## [Offloading task-results to S3/Database/etc](https://dask.discourse.group/t/offloading-task-results-to-s3-database-etc/4223)
**Author:** [@oxinabox](https://dask.discourse.group/u/oxinabox)\ **Replies:** 0\ **Last updated:** [September 14, 2026, 1:40am UTC](https://dask.discourse.group/t/offloading-task-results-to-s3-database-etc/4223 "2026-09-14T01:40:59Z")
When a worker dies, the scheduler re-shedules work to recreate its task-results based on what task-results are still available on the workers. Potentially this could go back a long way, especially if many worker die at … --- ## [Best practice for safely overwriting lazy-loaded data](https://dask.discourse.group/t/best-practice-for-safely-overwriting-lazy-loaded-data/2295)
**Author:** [@LucaMarconato](https://dask.discourse.group/u/LucaMarconato)\ **Replies:** 3\ **Last updated:** [August 19, 2026, 3:30pm UTC](https://dask.discourse.group/t/best-practice-for-safely-overwriting-lazy-loaded-data/2295 "2026-08-19T15:30:03Z")
Dear Dask developers, I am one of the core developers of a library for representing and manipulating spatial molecular data in Python: GitHub - scverse/spatialdata: An open and interoperable data framework for spatial om… --- ## [How to scatter when using map\_blocks()](https://dask.discourse.group/t/how-to-scatter-when-using-map-blocks/4192)
**Author:** [@arthur-e](https://dask.discourse.group/u/arthur-e)\ **Replies:** 1\ **Last updated:** [June 12, 2026, 12:06pm UTC](https://dask.discourse.group/t/how-to-scatter-when-using-map-blocks/4192 "2026-06-12T12:06:37Z")
I am trying to apply concurrency when computing trends on a (T x N) dask array. # Reshaping xarray dataset in geographic space to (T x N) arr = ds.values.reshape((len(YEARS), mask.size)).astype(np.float32) # Extracting… --- ## [How do get values as they're available on a map\_blocks call](https://dask.discourse.group/t/how-do-get-values-as-theyre-available-on-a-map-blocks-call/4181)
**Author:** [@odinsbane](https://dask.discourse.group/u/odinsbane)\ **Replies:** 6\ **Last updated:** [June 12, 2026, 11:48am UTC](https://dask.discourse.group/t/how-do-get-values-as-theyre-available-on-a-map-blocks-call/4181 "2026-06-12T11:48:35Z")
I’m trying to process a large chunk of data using map\_blocks but I don’t know how to access the data as it is read while processing in parallel. out = dask.array.map\_blocks( torchit, dtype="float32", chunks = chunks ) p… --- ## [Taking max projection of a large array](https://dask.discourse.group/t/taking-max-projection-of-a-large-array/2841)
**Author:** [@dub2s](https://dask.discourse.group/u/dub2s)\ **Replies:** 6\ **Last updated:** [June 11, 2026, 3:09pm UTC](https://dask.discourse.group/t/taking-max-projection-of-a-large-array/2841 "2026-06-11T15:09:50Z")
Hii everyone I have a large lightsheet imaging data in the form of large number of .tiff files. In the initial stages of analysis, I want to take maximum projection of the data along one axis. For this, I am loading the… --- ## [Best imshow-alternative for dask?](https://dask.discourse.group/t/best-imshow-alternative-for-dask/3819)
**Author:** [@khyll](https://dask.discourse.group/u/khyll)\ **Replies:** 4\ **Last updated:** [June 11, 2026, 3:00pm UTC](https://dask.discourse.group/t/best-imshow-alternative-for-dask/3819 "2026-06-11T15:00:30Z")
Is there any function or package that allows me to display a dask 2D-array faster than using the following (using .compute)? import dask import matplotlib.pyplot as plt arr\_dask = dask.array.random.randint((400, 400)) … --- ## [Lazy sum\_labels using map\_blocks with lazy index](https://dask.discourse.group/t/lazy-sum-labels-using-map-blocks-with-lazy-index/4079)
**Author:** [@martijnvandermarel](https://dask.discourse.group/u/martijnvandermarel)\ **Replies:** 5\ **Last updated:** [June 11, 2026, 2:58pm UTC](https://dask.discourse.group/t/lazy-sum-labels-using-map-blocks-with-lazy-index/4079 "2026-06-11T14:58:16Z")
First of all this might turn out to be more of an xarray question/issue, so apologies in advance. However it is very much related to dask and the bigger picture issue could be solved with dask. What am I trying to do? I… --- ## [How to improve the processing speed](https://dask.discourse.group/t/how-to-improve-the-processing-speed/3708)
**Author:** [@kjz1997](https://dask.discourse.group/u/kjz1997)\ **Replies:** 5\ **Last updated:** [June 11, 2026, 2:42pm UTC](https://dask.discourse.group/t/how-to-improve-the-processing-speed/3708 "2026-06-11T14:42:35Z")
Hello, as I need to handle very large matrices, there is a memory shortage error when using numpy. Therefore, I have chosen to use dask.array to store matrices, but I am very unfamiliar with dask.array. When processing d… --- ## [Tracking progress of logical groupings of tasks at execution time](https://dask.discourse.group/t/tracking-progress-of-logical-groupings-of-tasks-at-execution-time/4092)
**Author:** [@AdeelH](https://dask.discourse.group/u/AdeelH)\ **Replies:** 2\ **Last updated:** [June 11, 2026, 2:11pm UTC](https://dask.discourse.group/t/tracking-progress-of-logical-groupings-of-tasks-at-execution-time/4092 "2026-06-11T14:11:40Z")
I want to group tasks together in a dask execution graph and then track the progress of those groups when the graph is executing. For instance, given the following operations: # a dask-backed xr.DataArray data\_array: x… --- ## [A more efficient way to sample an array](https://dask.discourse.group/t/a-more-efficient-way-to-sample-an-array/4114)
**Author:** [@Nestor\_Sanchez](https://dask.discourse.group/u/Nestor_Sanchez)\ **Replies:** 2\ **Last updated:** [June 11, 2026, 1:17pm UTC](https://dask.discourse.group/t/a-more-efficient-way-to-sample-an-array/4114 "2026-06-11T13:17:11Z")
I have a large, chunked 3D zarr of geospatial data with dimensions (features, y, x), chunked over (y, x), and I want to sample of around 1M points on the (y, x) coords to compute some quantiles. This is a very large arra… --- ## [Error while using ndmeasure.area : Cannot block reusing for graphs](https://dask.discourse.group/t/error-while-using-ndmeasure-area-cannot-block-reusing-for-graphs/4167)
**Author:** [@dub2s](https://dask.discourse.group/u/dub2s)\ **Replies:** 1\ **Last updated:** [June 11, 2026, 12:17pm UTC](https://dask.discourse.group/t/error-while-using-ndmeasure-area-cannot-block-reusing-for-graphs/4167 "2026-06-11T12:17:44Z")
Hii, I am intending to use dask-image to preprocess a lightsheet microscopy data. To optimize various steps, I am working on a subvolume of shape (170,600,900). After performing segmentation and labeling (using dask\_ima… --- ## [Stuck at "Waiting for scheduler to run"](https://dask.discourse.group/t/stuck-at-waiting-for-scheduler-to-run/1446)
**Author:** [@dragospopa420](https://dask.discourse.group/u/dragospopa420)\ **Replies:** 20\ **Last updated:** [May 27, 2026, 9:15pm UTC](https://dask.discourse.group/t/stuck-at-waiting-for-scheduler-to-run/1446 "2026-05-27T21:15:27Z")
Hello, I’m trying to deploy a cluster on google cloud with dask\_cloudprovider.gcp. I’ve removed everything related to my project and I’m basically just creating the cluster. I see that the scheduler gets created but t… --- ## [The result of p2p shuffle drop\_duplicates is different than the result of tasks shuffle drop\_duplicates](https://dask.discourse.group/t/the-result-of-p2p-shuffle-drop-duplicates-is-different-than-the-result-of-tasks-shuffle-drop-duplicates/4195)
**Author:** [@Sam](https://dask.discourse.group/u/Sam)\ **Replies:** 0\ **Last updated:** [April 21, 2026, 4:19am UTC](https://dask.discourse.group/t/the-result-of-p2p-shuffle-drop-duplicates-is-different-than-the-result-of-tasks-shuffle-drop-duplicates/4195 "2026-04-21T04:19:07Z")
I have a huge list of string IDs, and they are stored as csv files. After read from csv as Dask dataframe, i ran these two calls. The results between these two calls are actually different. The result of drop\_duplicates… --- ## [Intermittent \`CommClosedError\` / worker–worker connection resets under load on HPC despite high bandwidth and increased timeouts](https://dask.discourse.group/t/intermittent-commclosederror-worker-worker-connection-resets-under-load-on-hpc-despite-high-bandwidth-and-increased-timeouts/4179)
**Author:** [@maneesh29s](https://dask.discourse.group/u/maneesh29s)\ **Replies:** 1\ **Last updated:** [February 23, 2026, 5:44am UTC](https://dask.discourse.group/t/intermittent-commclosederror-worker-worker-connection-resets-under-load-on-hpc-despite-high-bandwidth-and-increased-timeouts/4179 "2026-02-23T05:44:26Z")
Problem description I am running a Dask application on an HPC cluster (SLURM) processing ~200 GB of Zarr data across 3 nodes. My dask cluster creation logic spins up multiple dask workers per node, ensuring that each wor… --- ## [Writing a sort in Dask](https://dask.discourse.group/t/writing-a-sort-in-dask/3795)
**Author:** [@mppf](https://dask.discourse.group/u/mppf)\ **Replies:** 3\ **Last updated:** [February 19, 2026, 10:21am UTC](https://dask.discourse.group/t/writing-a-sort-in-dask/3795 "2026-02-19T10:21:37Z")
Hi Dask community! I’m doing some evaluation of different distributed programming models and as part of that I’m trying to implement a distributed radix sort with a bunch of different distributed programming tools as a w… --- ## [Updating documentation for scale method](https://dask.discourse.group/t/updating-documentation-for-scale-method/4170)
**Author:** [@uwagura](https://dask.discourse.group/u/uwagura)\ **Replies:** 0\ **Last updated:** [February 12, 2026, 3:05pm UTC](https://dask.discourse.group/t/updating-documentation-for-scale-method/4170 "2026-02-12T15:05:52Z")
Hello, I am a newish dask user that was confused by some of the documentation for the scale method to jobqueue Clusters. The “How this works” page currently says that For the full computation, you will then ask for a … --- ## [Enforce Blocking on Task Submission when Workers are Saturated](https://dask.discourse.group/t/enforce-blocking-on-task-submission-when-workers-are-saturated/4163)
**Author:** [@Ale\_dev](https://dask.discourse.group/u/Ale_dev)\ **Replies:** 3\ **Last updated:** [February 9, 2026, 10:52am UTC](https://dask.discourse.group/t/enforce-blocking-on-task-submission-when-workers-are-saturated/4163 "2026-02-09T10:52:45Z")
Description I am encountering a behavior where tasks submitted from within a worker (using worker\_client) are dispatched to the scheduler immediately, even when the target workers are already fully occupied or saturated … --- ## [Shared in-memory data for workers on same machine?](https://dask.discourse.group/t/shared-in-memory-data-for-workers-on-same-machine/329)
**Author:** [@pe224](https://dask.discourse.group/u/pe224)\ **Replies:** 6\ **Last updated:** [February 8, 2026, 4:10pm UTC](https://dask.discourse.group/t/shared-in-memory-data-for-workers-on-same-machine/329 "2026-02-08T16:10:11Z")
Ideally I’d like to leverage dask for distributing work between several machines as well as distributing it between multiple worker processes on each single machine. Each task is CPU-heavy and needs read-only access to … --- ## [Detect if \`Client\` spawned by the user is using threads or processes](https://dask.discourse.group/t/detect-if-client-spawned-by-the-user-is-using-threads-or-processes/4159)
**Author:** [@vpadulan](https://dask.discourse.group/u/vpadulan)\ **Replies:** 1\ **Last updated:** [February 8, 2026, 4:07pm UTC](https://dask.discourse.group/t/detect-if-client-spawned-by-the-user-is-using-threads-or-processes/4159 "2026-02-08T16:07:32Z")
Hi, TL;DR: I want to know how to detect whether a dask Client was created with processes=False or not, i.e. whether the Dask client is going to use thread scheduling or process scheduling. Keep reading for more context. … --- ## [Use lsf with dask with two workers classes](https://dask.discourse.group/t/use-lsf-with-dask-with-two-workers-classes/4154)
**Author:** [@vianneyl](https://dask.discourse.group/u/vianneyl)\ **Replies:** 2\ **Last updated:** [February 8, 2026, 4:01pm UTC](https://dask.discourse.group/t/use-lsf-with-dask-with-two-workers-classes/4154 "2026-02-08T16:01:49Z")
Hi, I am currently running a cluster on my local machine. It is made up of standard dask workers and dask-cuda workers (both instantiated from terminal, as well as scheduler). Basically it is equivalent to a Spec luster… --- ## [DataNode initialization / construction](https://dask.discourse.group/t/datanode-initialization-construction/4153)
**Author:** [@Jda](https://dask.discourse.group/u/Jda)\ **Replies:** 1\ **Last updated:** [February 8, 2026, 3:20pm UTC](https://dask.discourse.group/t/datanode-initialization-construction/4153 "2026-02-08T15:20:35Z")
hi all, i’m just getting started with Dask and trying to familiarize myself with how stuff works. i’ve started by looking at the unit tests, specifically test\_merge.py::test\_merge where it calls from\_pandas() to constru… --- ## [Request for Manufacturer obsolescence date/ EOL Date](https://dask.discourse.group/t/request-for-manufacturer-obsolescence-date-eol-date/4149)
**Author:** [@Aditya\_shah](https://dask.discourse.group/u/Aditya_shah)\ **Replies:** 1\ **Last updated:** [February 8, 2026, 3:15pm UTC](https://dask.discourse.group/t/request-for-manufacturer-obsolescence-date-eol-date/4149 "2026-02-08T15:15:37Z")
Dear Team, Could you please confirm the official obsolescence / end-of-support dates for the following components? Dask (0.18.1) Please also share: • Manufacturer current version for these above component. • The re… --- ## [Scheduler memory management](https://dask.discourse.group/t/scheduler-memory-management/4097)
**Author:** [@cbritogonzalez](https://dask.discourse.group/u/cbritogonzalez)\ **Replies:** 2\ **Last updated:** [January 21, 2026, 9:41am UTC](https://dask.discourse.group/t/scheduler-memory-management/4097 "2026-01-21T09:41:07Z")
Hello everyone, I have a question about how the dask scheduler manages memory internally, in a previous thread I got the ECSCluster up and running with the adaptive functionality turned on ECSCluster adaptivity - Distr… --- ## [Segment Fault due to Side Effect](https://dask.discourse.group/t/segment-fault-due-to-side-effect/4132)
**Author:** [@Karl\_Han](https://dask.discourse.group/u/Karl_Han)\ **Replies:** 3\ **Last updated:** [December 15, 2025, 4:42pm UTC](https://dask.discourse.group/t/segment-fault-due-to-side-effect/4132 "2025-12-15T16:42:06Z")
This is not a good example to demonstrate the segment fault triggered, which shows no useful log at all, but it still shows the side effect on each worker that executes tasks. The following code demonstrates that the si… --- ## [Zarr chunks locality](https://dask.discourse.group/t/zarr-chunks-locality/4139)
**Author:** [@orliac](https://dask.discourse.group/u/orliac)\ **Replies:** 1\ **Last updated:** [December 15, 2025, 4:22pm UTC](https://dask.discourse.group/t/zarr-chunks-locality/4139 "2025-12-15T16:22:55Z")
Hello. I’m seeking advice on how to control data locality when reading chunks from a zarr file. That is, I want to express the fact that all chunks to be processed by (distributed) worker i are loaded by worker i so that… --- ## [Dask-Gateway 2025/26](https://dask.discourse.group/t/dask-gateway-2025-26/4137)
**Author:** [@lyfo-dev](https://dask.discourse.group/u/lyfo-dev)\ **Replies:** 1\ **Last updated:** [December 15, 2025, 4:15pm UTC](https://dask.discourse.group/t/dask-gateway-2025-26/4137 "2025-12-15T16:15:48Z")
Hallo, I have scoured most of the docs related to deployments of distributed computing solutions, and it seems that Dask is the way to go. However, my setup would require to abstract Kubernetes away from the analysts. W… --- ## [Question about Dask array reduction and dask\_image ndmeasure](https://dask.discourse.group/t/question-about-dask-array-reduction-and-dask-image-ndmeasure/4140)
**Author:** [@Vincent](https://dask.discourse.group/u/Vincent)\ **Replies:** 1\ **Last updated:** [December 1, 2025, 9:00pm UTC](https://dask.discourse.group/t/question-about-dask-array-reduction-and-dask-image-ndmeasure/4140 "2025-12-01T21:00:36Z")
Hello everyone, I have been working on large 3D image processing with dask and dask\_image. The default dask\_image ndmeasure functions worked for me, though loading data into memory is a major bottleneck. Changing to a s… --- ## [Run dask inside dask](https://dask.discourse.group/t/run-dask-inside-dask/4141)
**Author:** [@vladidobro](https://dask.discourse.group/u/vladidobro)\ **Replies:** 1\ **Last updated:** [December 1, 2025, 1:00pm UTC](https://dask.discourse.group/t/run-dask-inside-dask/4141 "2025-12-01T13:00:15Z")
Hi! I have some code that uses satpy, which in turn uses xarray with dask for computations. The satpy reader works only with local files and insists on using xarray with dask arrays (I didn’t find an option to turn it o… --- ## [Xarray + Dask Memory Overflow When Processing Large NC Datasets - Chunking Not Working](https://dask.discourse.group/t/xarray-dask-memory-overflow-when-processing-large-nc-datasets-chunking-not-working/4134)
**Author:** [@longkaihao](https://dask.discourse.group/u/longkaihao)\ **Replies:** 2\ **Last updated:** [November 29, 2025, 7:15am UTC](https://dask.discourse.group/t/xarray-dask-memory-overflow-when-processing-large-nc-datasets-chunking-not-working/4134 "2025-11-29T07:15:13Z")
Hello everyone, I’m struggling with a persistent memory overflow issue when processing a large number of NetCDF files using Xarray and Dask. I’ve tried adjusting chunking strategies multiple times, but the memory usage k… [Next page](https://dask.discourse.group/latest.md?page=1)