Hello folks,
I’m trying to do a calculation of a 30 GB inside 4 clustered GPUs.
Even if I split this data into small chunks of 100 MB, the memory increase so much that it reports allocation issues.
The point is… How can I efficiently profile the GPU memory usage of my process? For further information, I’m using CuPy and Dask Arrays.
If I use only CPU and local memory, I could easily use the dask-memusage
plugin, but unfortunately, it does not work with GPUs.
I’m not using the dashboard because I’m running on a cluster that does not let me open ports externally.
Any thoughts and suggestions are welcome.