Trying to find out if there’s a way to have Dask record custom metrics and surface them some way to monitor pipeline-specific aspects. I’m imagining something like Apache Beam’s metrics where you can do something along the lines of beam.metrics.Counter('mynamespace', 'myvar').inc() across the various workers and they’re aggregated by the workflow executor, e.g. Dataflow’s Cloud Monitoring UI or, in an ideal world, Prometheus.
Does something like this exist in Dask?
I’ve done a bit of research in the documentation and also through the source code, but AFAIK, there is currently no such functionality in Dask.
You might use Actors to record some counters, but there is no way to expose them.
Dask also proposes a Prometheus endpoint, but I cannot see an easy way to expose custom metrics.
Finally, there is the Fine Performance Metrics mechanism, in which it seems you can add some custom things.
Anyway, your proposition sounds really interesting, I’m wondering what others have to say here. cc @jacobtomlinson @crusaderky.
Fine Performance Metrics do what you’re asking for