Page tree
Skip to end of metadata
Go to start of metadata


After an extensive evaluation period, we have chosen SLURM to be the resource manager and job scheduler for the new Pearcey cluster. We believe that Slurm is more suitable for effectively managing the compute resources we have on our compute clusters.

This reference guide provides information on migrating from Torque to Slurm.

See also the SLURM cheat sheet.


As Ruby is a NUMA system it is critical to co-locate cores and local memory. As such the batch system has a default memory per core enabled (12.8GB) and you should not request --mem. See the Requesting resources in Slurm guide for information about how to request resources for jobs on Ruby.

Common job commands


Submit a job

qsub <job script>
sbatch <job script>

Delete a job

qdel <job ID>
scancel <job ID>

Job status (all)


Job status (by job)

qstat <job ID>
squeue -j <job ID>

Job status (by user)

qstat -u <user>
squeue -u <user>

Job status (detailed)

qstat -f <job ID>
scontrol show job <job ID>

Show expected start time

showstart <job ID>
squeue -j <job ID> --start

Queue list / info

qstat -q [queue]
scontrol show partition [queue]

Node list

pbsnodes -a
scontrol show nodes

Node details

pbsnodes <node>
scontrol show node <node>

Hold a job

qhold <job ID>
scontrol hold <job ID>

Release a job

qrls <job ID>
scontrol release <job ID>

Cluster status

qstat -B

Start an interactive job

qsub -I <args>
sinteractive <args>

X forwarding

qsub -l -X <args>
salloc <args>
srun --pty <args>

Read stdout messages at runtime

qpeek <job ID>
No equivalent command / not needed.
Use the --output option instead.
Monitor cluster and jobs
Monitor or review a jobs resource usage 
sacct -j <job_num> --format JobID,jobname,NTasks,nodelist,CPUTime,ReqMem,MaxVMSize,Elapsed

Job submission options

Torque (qsub)
Slurm (sbatch)

Script directive


Job name

-N <name>


-q <queue>

Wall time limit

-l walltime=<hh:mm:ss>

Node count

-l nodes=<count>

Process count per node

-l ppn=<count>
core count (per process) 

Memory limit

-l vmem=<limit>
(in mega bytes - MB)

Minimum memory per processor

no equivalent

Request generic resource

(on the bragg-l accelerator cluster)

-l gpus=<count> 
-l mics=<count>

Request specific nodes

-l nodes=<node>[,node2[,...]]>
-w, --nodelist=<node>[,node2[,...]]>
-F, --nodefile=<node file>

Job array

-t <array indices>
-a <array indices>

Standard output file

-o <file path>
--output=<file path>
Standard error file
-e <file path>
--error=<file path>

Combine stdout/stderr to stdout

-j oe
--output=<combined out and err file path>

Copy environment

--export=ALL (default)

Copy environment variable

-v <variable[=value][,variable2=value2[,...]]>

Job dependency

-W depend=after:jobID[:jobID...]
-W depend=afterok:jobID[:jobID...]
-W depend=afternotok:jobID[:jobID...]
-W depend=afterany:jobID[:jobID...]

Request event notification

-m <events>

Note: require multiple mail-type requests for multiple events, such as:

#SBATCH --mail-type=BEGIN
#SBATCH --mail-type=END

Email address

-M <email address>
--mail-user=<email address>

Defer job until the specified time

-a <date/time>

Job environment

The  SLURM system will propagate the module environment of a users current environment (the environment of the shell from which a user calls sbatch) through to the worker nodes, with some exceptions noted in the following test. By default  SLURM does not source the files ~./bashrc or ~/.profile when requesting resources via sbatch (although it does when running sinteractive/ salloc ).  So, if you have a standard enviroment that you have set in either of these files or your current shell then you can do one of the following:

  1. Add the command #SBATCH --get-user-env to your job script (i.e. the module environment is propagated).

  2. Source the configuration file in your job script: 

    Sourcing your .bashrc file
    < #SBATCH statements >
    source ~/.bashrc
  3. You may want to remove the influence of any other current environment variables by adding #SBATCH --export=NONE to the script. This removes all set/exported variables and then acts as if #SBATCH --get-user-env has been added (module environment is propagated).

OpenMP can require a variable named OMP_NUM_THREADS to be set so as to specifiy the number of threads to create when code encounters an OpenMP code block, either in your own code or in a library function e.g. in MKL BLAS. An appropriate value for OMP_NUM_THREADS  can be obtained from the SLURM environment variable  $SLURM_CPUS_PER_TASK  that is set when  --cpus-per-task is specified in an sbatch script.

# Set the number of cores available per process
# if the $SLURM_CPUS_PER_TASK variable is not zero length
if [ ! -z  $SLURM_CPUS_PER_TASK ] ; then
	# $SLURM_CPUS_PER_TASK is a zero length
	# The default value is a core per task


Can extract from "sbatch --version".

Job name


Job ID


Batch or interactive


Batch server


Submit directory

Slurm jobs starts from the submit directory by default.

Submit host

Node file
A filename and path that lists the nodes a job has been allocated

Node list

The SLURM variable has a different format to the PBS one.
To get a list of nodes use:
"scontrol show hostnames $SLURM_JOB_NODELIST"

Job array index




Queue name


Number of nodes allocated


Number of processes


Number of processes per node


List of allocated GPUs


Requested tasks per node


Requested CPUs per task


Scheduling priority


Job user

Hostname$HOSTNAME$HOSTNAME == $SLURM_SUBMIT_HOST Unless a shell is envoked on an allocated resource, the HOSTNAME variable is propagated (copied) from the submit machine environmentsp will be the same on all allocated nodes.

Sample job script


#PBS -N testjob
#PBS -l walltime=2:00:00
echo "running job"
sleep 120
echo "bye"


#SBATCH --job-name="testjob"
#SBATCH --time=2:00:00
echo "running job"
sleep 120
echo "bye"
  • No labels