My goal is to process 1000 files at once on a SLURM cluster. I can perform this on 8 nodes each with 32 cores. But then trying with 16 nodes (each with 32 cores) almost in the end of the computation I receive an error message that I have too many open files (Too many open files: ‘/proc/PID/stat’ and…

Too many open files on SLURMCluster

scharlottej13 Split this topic January 11, 2022, 4:41pm 7

2 posts were merged into an existing topic: Task Scheduling for a single node level

Topic		Replies	Views
SLURMCluster on 64 nodes / Understanding Cluster scale method Distributed distributed	4	304	November 17, 2021
Parallelisation by multiprocessing not multithreading on SLURMCluster Distributed	1	321	April 23, 2022
Very low CPU utilization on SLURMCluster Distributed distributed , performance	3	1033	November 8, 2021
Dask distributed with very large queue Distributed	5	1073	November 13, 2022
Ensuring Each Dask Task Starts on a New SLURM Job with a Limit of 5 Concurrent Jobs Distributed distributed	2	201	October 27, 2023