My goal is to process 1000 files at once on a SLURM cluster. I can perform this on 8 nodes each with 32 cores. But then trying with 16 nodes (each with 32 cores) almost in the end of the computation I receive an error message that I have too many open files (Too many open files: ‘/proc/PID/stat’ and…

Too many open files on SLURMCluster

scharlottej13 Split this topic January 11, 2022, 4:41pm 7

2 posts were merged into an existing topic: Task Scheduling for a single node level

Topic		Replies	Views
Some questions about SLURMCluster dask-jobqueue	5	1398	March 6, 2023
Memory allocation always <= 4GiB for distributed SLURMCluster workers Distributed dask-jobqueue , worker , distributed	8	769	July 12, 2022
SLURMCluster on 64 nodes / Understanding Cluster scale method Distributed distributed	4	338	November 17, 2021
dask_jobqueue.SLURMCluster: multi-threaded workloads and the effect of setting "cores" Distributed dask-jobqueue , distributed	2	303	November 9, 2023
Can not run dask on cluster with LSF Distributed	3	200	January 17, 2024