Dask EC2 in Private Subnet

Distributed
1

Hi All,

I’m trying to get dask ec2 cluster working with my private subnet. currently, it looks like the api only allows for creating EC2 instances with a public IP. This is a bit of a security risk for us. I see there is a discussion: Allow EC2 cluster to be created without a public IP attached · Issue #284 · dask/dask-cloudprovider · GitHub

Is there any chance that could be looked at/prioritised ? and if not, what would I need to change to get this working ? I hacked it to try just replace the PublicIPAddress with PrivateIPAddress in daskcloudprovider/aws/ec2.py and it looked like the EC2 instance starts, however the docker container fails to start and my python script then site there waiting for the scheduler. From the log lines, it also looked like the scheduler tries to start on my private IP Address as well

Many thanks in advance

2

@davico8881 Welcome to Discourse!

Is there any chance that could be looked at/prioritised?

I’d encourage you to chime in on the issue directly to ask the maintainers! Since it has the “help wanted” label, if you’d be interested in working on it, maintainers may be able to guide you. :smile:

I’m not aware of workarounds (I’ll look into it though), but I believe Dask Gateway or hosted solutions can be some alternatives that can help.

3

Looks like the issue you linked to has been resolved! :tada: