# Optimal way to monitor GPU memory usage during distributed training (XGBoost)

**URL:** https://dask.discourse.group/t/optimal-way-to-monitor-gpu-memory-usage-during-distributed-training-xgboost/3998
**Category:** Distributed
**Created:** [June 27, 2025, 1:58pm UTC](https://dask.discourse.group/t/optimal-way-to-monitor-gpu-memory-usage-during-distributed-training-xgboost/3998 "2025-06-27T13:58:12Z")
**Posts on this page:** 1
**Showing post:** 2

<div class="post-metadata">

### Author: ![guillaumeeb](https://yyz1.discourse-cdn.com/flex035/user_avatar/dask.discourse.group/guillaumeeb/32/1613_2.png) [@guillaumeeb](https://dask.discourse.group/u/guillaumeeb)
#### Post date: [June 27, 2025, 2:44pm UTC](https://dask.discourse.group/t/optimal-way-to-monitor-gpu-memory-usage-during-distributed-training-xgboost/3998/2 "2025-06-27T14:44:04Z")

</div>

Hi @ap213, welcome to Dask Discourse forum!

First, I would like to be sure you are really launching computations on GPUs, are I don’t see any hints of that in your code. Are you configuring something, somewhere, to be sure that the code is running on GPUs? From the code I see, you are creating standard Dask array, so they would be held in server main memory and using CPUs, creating a LocalCUDACluster is not enough, but maybe you just didn’t put some part of the code.

To be more specific, you should use cupy or use it as a backend, as in [XGBoost example](https://xgboost.readthedocs.io/en/latest/python/dask-examples/gpu_training.html#sphx-glr-python-dask-examples-gpu-training-py):

```python
with Client(cluster) as client, dask.config.set({"array.backend": "cupy"}):

```

Next, or in the meantime, I would also check that GPUs are correctly used by using system tools like `nvidia-smi`. If you see some usage here, you should be able to get it from Python.

You can also use the Dask dashboard which [has gpu support if dask-cuda is installed](https://dask.discourse.group/t/how-to-monitor-gpu-usage-using-dashboard/2979/3).

---

_[View the full topic](https://dask.discourse.group/t/optimal-way-to-monitor-gpu-memory-usage-during-distributed-training-xgboost/3998)._