How can I print all remote dask-gateway-cluster workerlogs simultaneously? As mentioned in the post linked below?
Did you try client.get_worker_logs ?
However, you probably won’t get the Nannie’s logs, and maybe not all the stderr/stdout statements.
I think that you’ll have to issue kubectl commands on pods if you’re deploying Dask Gateway over Kubernetes.
Thanks alot, works from within a notebook.
It takes also an argument called nanny, which might be what I need to use, according to the post linked.
Edit: Ignore this, it is solved with client.get_worker_logs(nanny=True).
@guillaumeeb I am not using Kubernetes. Just in case get_worker_logs happens to not be the right log I should look at, is there a way I can get the log mentioned (stderr/stdout-statements) from inside a notebook, without kubectl?
Not sure if that log is crucial to the troubleshooting of the Couldn’t gather keys, rescheduling-problem, but would like to have everything in case that is where the issue is described.
Which backend does your Dask Gateway cluster setup use? I don’t think you can get all the stdout/stderr logs from inside a notebook.
1 Like
Okay, I dont use Kubernetes, but maybe I misunderstood your previous question, beause it is probably used in the deployment I am using (I think, but I also have a lack of knowledge of Dask Gateway backend). I am running computations on this Google Cloud deployment.
Yes, this is a Kubernetes deployment, but anyway, your problem is solved I think, so all good!
1 Like