Hi I tried to use delayed with dataframe, and i want to keep all data in workers rather than pulling into my local machine, because data is huge(more than 30GB, and are read from parquet by dd.read_parquet). I wrote the code, and generally i want to keep data in workers and i need to operate on Pand…

I suppose can use map_partitions :slight_smile:

map_partitions is certainly simpler if it fits your need! But the first provided code should work too if you ever need it.

Help to check my delayed methods with Dask dataframe

Sam April 1, 2024, 1:34am 2

I suppose can use map_partitions

Topic		Replies	Views
Question: if I am mixing dask.delayed functions and using dask dataframes, are there any caveats to be aware of? Dask DataFrame delayed	5	786	August 21, 2023
Dask.delayed and custom workflows Distributed kubernetes , delayed , distributed	2	333	February 21, 2024
Delayed argument in map_partition Dask DataFrame delayed	4	117	May 5, 2025
How to handle a Dask DF in multiple modules? Dask DataFrame	6	602	February 8, 2023
@delayed(nout=2) equivalent in dd.from_map? Dask DataFrame	5	60	November 22, 2024