Automatically spinning EC2 instances for scheduler and worker

Deploying Dask
1

I am trying to learn more by experimenting with dask at the lowest level possible. I am trying to carry on the instantiation of EC2 machines via boto3 in python and linking them together without using dask.cloudprovider or coiled.

For this experiment I set an image containing all my requirements and also dask[complete].

I was able to launch successfully dask scheduler+workers connecting via ssh to each node and calling dask scheduler and dask worker <scheduler_ip>:8786 . What I am struggling right now is with the right approach to automatically launch upon start of the machine the commands for dask scheduler and dask worker.

I tried so far two approaches (consider for example only “dask worker” launch):

  1. Adding a cronjob line of the type

@reboot dask_start.sh
where dask_start.sh has the call for init the dask worker.

  1. Passing the command via userData to the ec2 instance:
    Something along the lines:
#!/bin/bash 
/home/ubuntu/.local/bin/dask worker 172.31.43.110:8786

Any ideas on how to achieve this automatically instantiation of the worker/scheduler?

Thank you

2

Another option that I was wondering to be possible was to run remotelly in AWS EC2 via SSM protocol the dask scheduler and dask worker command but havent tried this approach yet.

3

Create a systemd unit maybe?