I’m seeking for some advice on how to recompute reported bandwidths between pairs of workers in the performance report. What I would like to see are all the transfers accounted for in their computations. For now I don’t see how to derive such metrics from the information reported in workers’ log.
Hi @orliac, welcome to Dask community!
Cloud you be more precise on the information you want to retrieve? Is it something that you can see on the Dashboard but are unable to get by yourself, and if so which part?
Workers’ log don’t give much information about this, however, you can find plenty of statistics through diverse means:
- Using dedicated API for diagnostics. Not sure if there is what you are after, but you can almost get the dashboard.
- Using Prometheus, for example
dask_worker_transfer_outgoing_bytes, - Or directly by inspecting Python object (which is what does the Dashboard and Prometheus). See an example from Dashboard code here.
I can also give some simple code sample:
from distributed import Client
# create client
client = Client(n_workers=2, threads_per_worker=2, memory_limit='2GiB')
workers = client.cluster.scheduler.workers.values()
for ws in workers:
print(ws.metrics['transfer'])
gives:
{'incoming_bytes': 0, 'incoming_count': 0, 'incoming_count_total': 0, 'outgoing_bytes': 0, 'outgoing_count': 0, 'outgoing_count_total': 0}
{'incoming_bytes': 0, 'incoming_count': 0, 'incoming_count_total': 0, 'outgoing_bytes': 0, 'outgoing_count': 0, 'outgoing_count_total': 0}