Print dask-gateway-cluster workers logs, from a jupyter lab notebook

How can I print all remote dask-gateway-cluster workerlogs simultaneously? As mentioned in the post linked below?

Did you try client.get_worker_logs ?

However, you probably won’t get the Nannie’s logs, and maybe not all the stderr/stdout statements.

I think that you’ll have to issue kubectl commands on pods if you’re deploying Dask Gateway over Kubernetes.

Thanks alot, works from within a notebook.

It takes also an argument called nanny, which might be what I need to use, according to the post linked.

Edit: Ignore this, it is solved with client.get_worker_logs(nanny=True).

@guillaumeeb I am not using Kubernetes. Just in case get_worker_logs happens to not be the right log I should look at, is there a way I can get the log mentioned (stderr/stdout-statements) from inside a notebook, without kubectl?

Not sure if that log is crucial to the troubleshooting of the Couldn’t gather keys, rescheduling-problem, but would like to have everything in case that is where the issue is described.

Which backend does your Dask Gateway cluster setup use? I don’t think you can get all the stdout/stderr logs from inside a notebook.

1 Like

Okay, I dont use Kubernetes, but maybe I misunderstood your previous question, beause it is probably used in the deployment I am using (I think, but I also have a lack of knowledge of Dask Gateway backend). I am running computations on this Google Cloud deployment.

Yes, this is a Kubernetes deployment, but anyway, your problem is solved I think, so all good!

1 Like