Hi @vigneshn1997, this seems like a follow-up to your previous post, thanks for bringing it up again! Dask has a number of ways to read and process data in parallel, can you provide more information on what kind of data you’re using and where it is stored? In the meantime, you may find this example is similar to your workflow and there’s also this reference for connecting to remote data.
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
|
Reading data (and image data) from HDFS for training
|
2 | 223 | September 28, 2023 | |
| Use multiple workers to load data into dask.array | 7 | 966 | April 9, 2022 | |
| Dividing data among workers and downloading data local to a worker | 3 | 396 | February 11, 2022 | |
| Issue in Parallel row preprocessing with Dask | 2 | 496 | August 6, 2022 | |
| Shared in-memory data for workers on same machine? | 3 | 878 | February 7, 2022 |