Skip to content

Main Slurm Commands

Submit Jobs

There are three ways of submitting jobs with slurm, using either sbatch, srun or salloc:

### /!\ Adapt <partition>, <qos>, <account> and <command> accordingly
sbatch -p <partition> [--qos <qos>] [-A <account>] [...] <path/to/launcher.sh>

### /!\ Adapt <partition>, <qos>, <account> and <command> accordingly
srun -p <partition> [--qos <qos>] [-A <account>] [...] ---pty bash
srun is also to be using within your launcher script to initiate a job step.

# Request interactive jobs/allocations
### /!\ Adapt <partition>, <qos>, <account> and <command> accordingly
salloc -p <partition> [--qos <qos>] [-A <account>] [...] <command>

sbatch

sbatch is used to submit a batch launcher script for later execution, corresponding to batch/passive submission mode. The script will typically contain one or more srun commands to launch parallel tasks. Upon submission with sbatch, Slurm will:

  • allocate resources (nodes, tasks, partition, constraints, etc.)
  • runs a single copy of the batch script on the first allocated node
    • in particular, if you depend on other scripts, ensure you have refer to them with the complete path toward them.

When you submit the job, Slurm responds with the job's ID, which will be used to identify this job in reports from Slurm.

# /!\ ADAPT path to launcher accordingly
$ sbatch <path/to/launcher>.sh
Submitted batch job 864933

srun

srun is used to initiate parallel job steps within a job OR to start an interactive job Upon submission with srun, Slurm will:

  • (eventually) allocate resources (nodes, tasks, partition, constraints, etc.) when run for interactive submission
  • launch a job step that will execute on the allocated resources.

A job can contain multiple job steps executing sequentially or in parallel on independent or shared resources within the job's node allocation.

salloc

salloc is used to allocate resources for a job in real time. Typically this is used to allocate resources (nodes, tasks, partition, etc.) and spawn a shell. The shell is then used to execute srun commands to launch parallel tasks.

Interactive jobs: si*

You should use the helper functions si, si-gpu, si-bigmem to submit an interactive job.

For more details, see interactive jobs.

Collect Job Information

Command Description
sacct [-X] -j <jobid> [...] display accounting information on jobs.
scontrol show [...] view and/or update system, nodes, job, step, partition or reservation status
seff <jobid> get efficiency metrics of past job
smap graphically show information on jobs, nodes, partitions
sprio show factors that comprise a jobs scheduling priority
squeue [-u $(whoami)] display jobs[steps] and their state
sstat show status of running jobs.

squeue

You can view information about jobs located in the Slurm scheduling queue (partition/qos), eventually filter on specific job state (R:running /PD:pending / F:failed / PR:preempted) with squeue:

$ squeue [-u <user>] [-p <partition>] [---qos <qos>] [--reservation <name>] [-t R|PD|F|PR]

To quickly access your jobs, you can simply use sq

Live job statistics

You can use the scurrent (for current interactive job) or (more generally) scontrol show job <jobid> to collect detailed information for a running job.

scontrol show job <jobid>
$  scontrol show job 2166371
JobId=2166371 JobName=bash
   UserId=<login>(<uid>) GroupId=clusterusers(666) MCS_label=N/A
   Priority=12741 Nice=0 Account=ulhpc QOS=debug JobState=RUNNING Reason=None
   [...]
   SubmitTime=2020-12-07T22:08:25 EligibleTime=2020-12-07T22:08:25
   StartTime=2020-12-07T22:08:25 EndTime=2020-12-07T22:38:25
   [...]
   WorkDir=/mnt/irisgpfs/users/<login>

Past job statistics: slist, sreport

Use the slist helper for a given job:

# /!\ ADAPT <jobid> accordingly
$ slist <jobid>
# sacct -j <JOBID> --format User,JobID,Jobname%30,partition,state,time,elapsed,\
#              MaxRss,MaxVMSize,nnodes,ncpus,nodelist,AveCPU,ConsumedEnergyRaw
# seff <jobid>

You can also use sreport o generate reports of job usage and cluster utilization for Slurm jobs. For instance, to list your usage in CPU-hours since the beginning of the year:

$ sreport -t hours cluster UserUtilizationByAccount Users=$USER  Start=$(date +%Y)-01-01
--------------------------------------------------------------------------------
Cluster/User/Account Utilization 2021-01-01T00:00:00 - 2021-02-13T23:59:59 (3801600 secs)
Usage reported in CPU Hours
----------------------------------------------------------------------------
  Cluster     Login     Proper Name                Account     Used   Energy
--------- --------- --------------- ---------------------- -------- --------
     iris   <login>          <name> <firstname>.<lastname>    [...]
     iris   <login>          <name>      project_<acronym>    [...]

Job efficiency

seff

Use seff to double check a past job CPU/Memory efficiency. Below examples should be self-speaking:

$ seff 2171749
Job ID: 2171749
Cluster: iris
User/Group: <login>/clusterusers
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 28
CPU Utilized: 41-01:38:14
CPU Efficiency: 99.64% of 41-05:09:44 core-walltime
Job Wall-clock time: 1-11:19:38
Memory Utilized: 2.73 GB
Memory Efficiency: 2.43% of 112.00 GB
$ seff 2117620
Job ID: 2117620
Cluster: iris
User/Group: <login>/clusterusers
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 16
CPU Utilized: 14:24:49
CPU Efficiency: 23.72% of 2-12:46:24 core-walltime
Job Wall-clock time: 03:47:54
Memory Utilized: 193.04 GB
Memory Efficiency: 80.43% of 240.00 GB
$ seff 2138087
Job ID: 2138087
Cluster: iris
User/Group: <login>/clusterusers
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 64
CPU Utilized: 87-16:58:22
CPU Efficiency: 86.58% of 101-07:16:16 core-walltime
Job Wall-clock time: 1-13:59:19
Memory Utilized: 1.64 TB
Memory Efficiency: 99.29% of 1.65 TB

This illustrates a very bad job in terms of CPU/memory efficiency (below 4%), which illustrate a case where basically the user wasted 4 hours of computation while mobilizing a full node and its 28 cores.

$ seff 2199497
Job ID: 2199497
Cluster: iris
User/Group: <login>/clusterusers
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 28
CPU Utilized: 00:08:33
CPU Efficiency: 3.55% of 04:00:48 core-walltime
Job Wall-clock time: 00:08:36
Memory Utilized: 55.84 MB
Memory Efficiency: 0.05% of 112.00 GB
This is typical of a single-core task can could be drastically improved via GNU Parallel.

Note however that demonstrating a CPU good efficiency with seff may not be enough! You may still induce an abnormal load on the reserved nodes if you spawn more processes than allowed by the Slurm reservation. To avoid that, always try to prefix your executions with srun within your launchers. See also Specific Resource Allocations.

susage

Use susage to check your past jobs walltime accuracy (Timelimit vs. Elapsed)

$ susage -h
Usage: susage [-m] [-Y] [-S YYYY-MM-DD] [-E YYYT-MM-DD]
  For a specific user (if accounting rights granted):    susage [...] -u <user>
  For a specific account (if accounting rights granted): susage [...] -A <account>
Display past job usage summary

Official sacct command

Alternatively, you can use sacct (use sacct --helpformat to get the list of) for COMPLETED or TIMEOUT jobs (see Job State Codes).

using sacct -X -S <start> [...] --format [...],time,elapsed,[...]

ADAPT -S <start> and -E <end> dates accordingly - Format: YYYY-MM-DD. hint: $(date +%F) will return today's date in that format, $(date +%Y) return the current year, so the below command will list your completed (or timeout jobs) since the beginning of the month:

$ sacct -X -S $(date +%Y)-01-01 -E $(date +%F) --partition batch,gpu,bigmem --state CD,TO --format User,JobID,partition%12,qos,state,time,elapsed,nnodes,ncpus,allocGRES
     User        JobID    Partition        QOS      State  Timelimit    Elapsed   NNodes      NCPUS    AllocGRES
--------- ------------ ------------ ---------- ---------- ---------- ---------- -------- ---------- ------------
 <login> 2243517             batch     normal    TIMEOUT 2-00:00:00 2-00:00:05        4        112
 <login> 2243518             batch     normal    TIMEOUT 2-00:00:00 2-00:00:05        4        112
 <login> 2244056               gpu     normal    TIMEOUT 2-00:00:00 2-00:00:12        1         16        gpu:2
 <login> 2246094               gpu       high    TIMEOUT 2-00:00:00 2-00:00:29        1         16        gpu:2
 <login> 2246120               gpu       high  COMPLETED 2-00:00:00 1-02:18:00        1         16        gpu:2
 <login> 2247278            bigmem     normal  COMPLETED 2-00:00:00 1-05:59:21        1         56
 <login> 2250178             batch     normal  COMPLETED 2-00:00:00   10:04:32        1          1
 <login> 2251232               gpu     normal  COMPLETED 1-00:00:00   12:05:46        1          6        gpu:1

Platform Status

sinfo

sinfo allow to view information about partition status (-p <partition>), problematic nodes (-R), reservations (-T), eventually in a summarized form (-s),

sinfo [-p <partition>] {-s | -R | -T |...}

We are providing a certain number of helper functions based on sinfo:

Command Description
nodelist List available nodes
allocnodes List currently allocated nodes
idlenodes List currently idle nodes
deadnodes List dead nodes per partition (hopefully none ;))
sissues List nodes with issues/problems, with reasons
sfeatures List available node features

Cluster, partition and QOS usage stats

We have defined several custom ULHPC Slurm helpers defined in /etc/profile.d/slurm.sh to facilitate access to account/parition/qos/usage information. They are listed below.

Command Description
acct <name> Get information on user/account holder <name> in Slurm accounting DB
irisstat, aionstat report cluster status (utilization, partition and QOS live stats)
listpartitionjobs <part> List jobs (and current load) of the slurm partition <part>
pload [-a] i/b/g/m Overview of the Slurm partition load
qload [-a] <qos> Show current load of the slurm QOS <qos>
sbill <jobid> Display job charging / billing summary
sjoin [-w <node>] join a running job
sassoc <name> Show Slurm association information for <name> (user or account)
slist <jobid> [-X] List statistics of a past job
sqos Show QOS information and limits
susage [-m] [-Y] [...] Display past job usage summary

Updating jobs

Command Description
scancel <jobid> cancel a job or set of jobs.
scontrol update jobid=<jobid> [...] update pending job definition
scontrol hold <jobid> Hold job
scontrol resume <jobid> Resume held job

The scontrol command allows certain charactistics of a job to be updated while it is still queued (i.e. not running ), with the syntax scontrol update jobid=<jobid> [...]

Important

Once the job is running, most changes requested with scontrol update jobid=[...] will NOT be applied.

Change timelimit

# /!\ ADAPT <jobid> and new time limit accordingly
scontrol update jobid=<jobid> timelimit=<[DD-]HH:MM::SS>

Change QOS or Reservation

# /!\ ADAPT <jobid>, <qos>, <resname> accordingly
scontrol update jobid=<jobid> qos=<qos>
scontrol update jobid=<jobid> reservationname=<resname>

Change account

If you forgot to specify the expected project account:

# /!\ ADAPT <jobid>, <account> accordingly
scontrol update jobid=<jobid> account=<account>

The new account must be eligible to run the job. See Account Hierarchy for more details.

Hold and Resume jobs

Prevent a pending job from being started:

# /!\ ADAPT <jobid>  accordingly
scontrol hold <jobid>

Allow a held job to accrue priority and run:

# /!\ ADAPT <jobid>  accordingly
scontrol release <jobid>

Cancel jobs

Cancel a specific job:

# /!\ ADAPT <jobid> accordingly
scancel <jobid>
Cancel all jobs owned by a user (you)

scancel -u $USER
This only applies to jobs which are associated with your accounts.


Last update: December 1, 2022