Workers on private network, scheduler on a different network - how to make "gather" step work?

[kreczko]

Thank you for your replies!

@guillaumeeb

To be honnest, I’m even surprised it goes this far! Do you happen to have some Error stacktrace to share?

@crusaderky

I too am surprised you got that far.

Given that the network communication flow is a black box, anything is possible ;).
Once a connection is established from the worker side, the scheduler and workers can communicate in both directions (TCP). However, once the connection is dropped, the scheduler has no way to contact the workers.

Does anyone know how the communication in Dask works (in the form of an Alice ↔ Bob graph)?

I had a look at the gateway proxy code - in theory, if there is a way to keep the connections until the cluster is shut down, this issue can be fixed.

I’ve also started reading up on a related solution for a different system: the HTCondor Connection Broker:

This is instantiated by the collector to provide connection brokering
service. Daemons (called “targets”) register with the server and keep
a TCP connection to it to exchange messages with the server. Clients
send requests to the server to get a target daemon to do a reverse
connect to the client.

Short version: Provide a server with 1 open port and it will establish connections between nodes that cannot talk to each other otherwise. TCP magic.