I have some code in Python that reads in several TIFF images, does some data processing, and then writes an output TIFF to disk. The output file is very large and writing it to disk takes a long time as the IO is done in serial.
Does dask support parallel IO operations? Using dask, can I have multiple workers write to different sections of a single file simultaneously?
Also, using dask, is it possible to distribute IO operations across the nodes in a cluster?
Hi @Wombat, welcome to Dask community!
I’m not aware of any Python library that support parallel IO on TIFF files, but I’d like to be proved wrong, and that could be some game changer!
Since Dask use Python processes, I don’t see that can be done.
It is, but again not in the same file. The workaround here is to create several TIFF files, and maybe create a VRT file to have a virtual dataset stacking all the files one way or another.
Yes, parallel access to TIFF is possible and normal. In particular, you should read up on “cloud optimized geoTIFF (CoG)”, which is has chunking designed with this in mind. As the name suggests, CoG is mainly used in earth sciences.
You may want to check out how xarray handles TIFF files (it uses rasterio, dask example here), that’s possibly the best analysis package for this kind of thing. It can transparently use dask and concat/stack your images if they have appropriate coordinates information. I don’t know, but I expect it uses a lock for writing TIFFs across a cluster.
See also: tifffile, which can extract the buffer locations in a TIFF, and can be used by kerchunk for making virtual zarr datasets out of tiffs.
@martindurant, I’m a bit curious about what you say. If I totally agree that parallel reading of TIFF file is doable in Python, I’m a bit more sceptical about parallel writes. I mean it in the sense of MPI-IO like tools, that allow real concurrent writes on a single file in a distributed way.
To the question
I believe the answer is no, isn’t that right?
It’s nice to have a mechanism already handling it through lock though, but the performances must be affected.
The rioxarray example I linked suggests that you can have multiple workers “involved” in writing to a single file, but there is a lock, and I don’t know how it works in practice; nor do I know whether the xarray API uses this code path.
So you could indeed write to separate files (as you suggested) and later use xarray (or kerchunk) to form a logical dataset over the multiple output files.