Hi @vigneshn1997, this seems like a follow-up to your previous post, thanks for bringing it up again! Dask has a number of ways to read and process data in parallel, can you provide more information on what kind of data you’re using and where it is stored? In the meantime, you may find this example is similar to your workflow and there’s also this reference for connecting to remote data.
Related Topics
Topic | Replies | Views | Activity | |
---|---|---|---|---|
Dask Arrays with TensorFlow | 3 | 771 | August 5, 2022 | |
Memoizing External to Cluster Table Reads | 6 | 63 | December 5, 2023 | |
Optimising Dask computations (memory implications and communication overhead) | 6 | 151 | October 12, 2023 | |
Dask not distributing reading of parquet file? | 1 | 1118 | April 6, 2023 | |
Is this a fair benchmarking approach? | 4 | 192 | June 8, 2022 |