Tracking and Storing Memory Usage Per Task

Promtheus endpoints are just an HTTP endpoint with a specific format. Of course the most common way to collect that data is with Prometheus itself, but you could just peridically curl the endpoint and store the data however you like.

Make sure you have the prometheus_client package installed and then access <dask scheduler IP>:8787/metrics.

You could add more metrics via a scheduler plugin. We don’t have any documentation on this so you will need to look at how the current metrics are implemented.