I am trying to learn more by experimenting with dask at the lowest level possible. I am trying to carry on the instantiation of EC2 machines via boto3 in python and linking them together without using dask.cloudprovider or coiled.
For this experiment I set an image containing all my requirements and also dask[complete].
I was able to launch successfully dask scheduler+workers connecting via ssh to each node and calling
dask scheduler and
dask worker <scheduler_ip>:8786 . What I am struggling right now is with the right approach to automatically launch upon start of the machine the commands for dask scheduler and dask worker.
I tried so far two approaches (consider for example only “dask worker” launch):
- Adding a cronjob line of the type
where dask_start.sh has the call for init the dask worker.
- Passing the command via userData to the ec2 instance:
Something along the lines:
#!/bin/bash /home/ubuntu/.local/bin/dask worker 172.31.43.110:8786
Any ideas on how to achieve this automatically instantiation of the worker/scheduler?