ULHPC Slurm QOS 2.0¶
Quality of Service or QOS is used to constrain or modify the characteristics that a job can have. This could come in the form of specifying a QoS to request for a longer run time or a high priority queue for a given job.
To select a given QOS with a Slurm command, use the
--qos <qos> option:
srun|sbatch|salloc|sinfo|squeue... [-p <partition>] --qos <qos> [...]
The default QoS of your jobs depends on your account and affiliation.
--qos <qos> directive does not need to be set for most jobs
We favor in general cross-partition QOS, mainly tied to priority level
A special preemptible QOS exists for best-effort
jobs and is named
List QOS Limits¶
sqos utility function to list the existing QOS limits.
List current ULHPC QOS limits with
$ sqos \# sacctmgr show qos format="name%20,preempt,priority,GrpTRES,MaxTresPerJob,MaxJobsPerUser,MaxWall,flags" Name Preempt Priority GrpTRES MaxTRES MaxJobsPU MaxWall Flags -------------------- ---------- ---------- ------------- ------------- --------- ----------- -------------------- normal besteffort 100 50 DenyOnLimit besteffort 1 100 NoReserve low besteffort 10 2 DenyOnLimit high besteffort 200 50 DenyOnLimit urgent besteffort 1000 100 DenyOnLimit debug besteffort 150 node=8 10 DenyOnLimit long besteffort 100 node=6 node=2 4 14-00:00:00 DenyOnLimit,Partiti+
What are the possible limits set on ULHPC QOS?
At the QOS level, the following elements are composed to define the resource limits for our QOS:
- Limits on Trackable RESources (TRES - a resource (cpu,node,etc.) tracked for usage or used to enforce limits against), in particular:
GrpTRES: The total count of TRES able to be used at any given time from jobs running from the QOS.
- If this limit is reached new jobs will be queued but only allowed to run after resources have been relinquished from this group.
MaxTresPerJob: the maximum size in TRES (cpu,nodes,...) any given job can have from the QOS
MaxJobsPerUser: The maximum number of jobs a user can have running at a given time
MaxWall[DurationPerJob]= The maximum wall clock time any individual job can run for in the given QOS.
As explained in the Limits section, there are basically three layers of Slurm limits, from least to most priority:
- account associations: Root/Cluster -> Account (ascending the hierarchy) -> User
- Job/Partition QOS